Many of us that work with data will find requests and demands to import or export data at some point. Plenty of us have regular processes that perform these actions, and we may regularly troubleshoot or enhance these activities. In fact, I know some people have a full time, or nearly full time, position just dealing with ETL operations.
Working with data in disparate formats and the myriad of inconsistencies even when formats are known is a challenge. Integration Services is a useful tool, but many us find that we need to pre or post process data separate from a simple import or export. Some of us may prefer using T-SQL or other languages, such as R or Python, to process data rather than programming SSIS. It seems that I often find that every client wants a slightly different format or change to their data that a simple query export won't handle.
These days, as we add in Machine Learning and other downstream processing activities, it seems that there is more and more of a need to process data beyond imports and exports. After all, it seems that the majority of the time in any ML project is spent preparing and transforming data. In addition, in Article 15 of the GDPR, there is language that notes a data subject has the right to request a copy of the data relating to them when it is being processed by an organization. I don't know how often someone will want to get data about themselves or their organization, but I'm sure it will happen more than it happens today.
I think this means I'll need to brush up on ETL skills, perhaps to ensure I can easily extract out a copy of an individual's data. In fact, I probably should compile some scripts now to ensure I can let someone know what we information keep at SQLServerCentral that would fall under GDPR. I think it's just email addresses, but I could be wrong.
When a SQL Server database is operating smoothly and performing well, there is no need to be particularly aware of the transaction log, beyond ensuring that every database has an appropriate backup regime and restore plan in place. When things go wrong, however, a DBA's reputation depends on a deeper understanding of the transaction log, both what it does, and how it works. More »
SQL Server Reporting Services has been the go to reporting solution for SQL Server for almost a decade now. In many organizations, the ease of development, web front end and security make SSRS the tool of choice. More »
(2018-Feb-10) A few days ago Microsoft provided another update for its desktop version of the Power BI data analytical tool... More »
Question of the Day
Today's Question (by Steve Jones):
I've got this data set:
rank player.name year2017 yards2017
1 1 Tom Brady 2017 4577
2 2 Philip Rivers 2017 4515
3 3 Matthew Stafford 2017 4446
4 4 Drew Brees 2017 4334
5 5 Ben Roethlisberger 2017 4251
I want to add a column to track how many yards each person is trailing the leader. How can I add a column to this data set and populate it with the number of yards behind the leader?
Think you know the answer? Click here, and find out if you are right.
We keep track of your score to give you bragging rights against your peers.
This question is worth
1 point in this category: R Language.
We'd love to give you credit for your own question and answer.
To submit a QOTD, simply log in to the
Design and configure SQL Server instances and databases in support of high-throughput applications that are mission-critical and provide consistent response times in the face of variations in user numbers and query volumes. Learn to configure SQL Server and design your databases to support a given instance and workload.
Pick up your copy of this great book today at Amazon today.
Yesterday's Question of the Day
(by Steve Jones):
I want to set some session key-value pairs using sp_set_session_context. Since this might be used in a number of places in my application, I'm concerned that developers might set too many pairs. What is the maximum amount of data I can store with this feature?
Looking to sort two queries by Date, Time
LTRIM(RIGHT(CONVERT(varchar(6), .,100),7)) AS
--, FORMAT(., 'MM-dd-yyyy') AS
, CONVERT(varchar(10),REPLACE(CONVERT(varchar(10), .,101),'/','-')) AS
This newsletter was sent to you because you signed up at SQLServerCentral.com.
Feel free to forward this to any colleagues that you think might be interested.
If you have received this email from a colleague, you can register to receive it here.