SQLServerCentral / Article Discussions / Article Discussions by Author / Discuss content posted by Hugo Kornelis / Reclaiming freed space / Latest Posts

RE: Reclaiming freed space

dstemate — Wed, 18 Nov 2009 10:31:59 GMT

I thought that truncate will do the job. Now I know it does not. Good question!

RE: Reclaiming freed space

SQLRNNR — Wed, 26 Aug 2009 09:03:52 GMT

Hugo and Hakan, both bring up good points. There may be justifiable cases to not have a clustered index on a table - in an ideal world where one has thought long and hard about the decision. Besides the ability to rebuild indexes online, or defrag the indexes in the table; we also have the added benefit of being able to more easily move tables to new filegroups - should we desire or the need arise. Too often it is a novice at database design creating new tables or even databases on the whole, and oftentimes all indexes are overlooked as well as keys of any sort (PK or FK). And agreed, it should be evaluated by the team (dev and dba) whether to use a clustered index in place of the non-clustered index - just in case there was a valid reason to just use non-clustered indexing.:hehe:

RE: Reclaiming freed space

hakan.winther — Wed, 26 Aug 2009 07:04:05 GMT

[quote]The answer option with a clustered index included dropping of same because it had to be assumed that the schema should remain unchanged. In an idea world, a table with no clustered index doesn't just happen; someone has thought about it long and hard, discussed it with a colleague, and then made the decision to not have a clustered index on the table. Such a decision should not be unmade on a wimp. Though I do of course know that this ideal world often has little to no resemblence of reality. ;)[/quote]I agree, you should think long and hard about the decision to not have a clustered index, but sadly, thats not true in many cases I have seen in reality. But even worse, many developers uses the management studio to design the tables and just click the "Primary key" button and create the table without any thought at all about if the Primary key is the best clustered index. I think Microsoft have made it too easy to design and develop a database, so everyone think they can do it. :(

RE: Reclaiming freed space

Hugo Kornelis — Tue, 25 Aug 2009 13:48:47 GMT

[quote][b]CirquedeSQLeil (8/25/2009)[/b][hr]This was a good question. I would have initially ticked the truncate table option but ran some testing and research first. Though accurate, I am not sure I agree with dropping the clustered index - I think I would rather drop the nonclustered index and leave a clustered index on the table.Thanks for the good question.[/quote]Hi Jason,Thanks for the kind words.The answer option with a clustered index included dropping of same because it had to be assumed that the schema should remain unchanged. In an idea world, a table with no clustered index doesn't just happen; someone has thought about it long and hard, discussed it with a colleague, and then made the decision to not have a clustered index on the table. Such a decision should not be unmade on a wimp. Though I do of course know that this ideal world often has little to no resemblence of reality. ;)In the large majority of cases, tables should have a clustered index. There are exceptions to this rules, but they should be a small minority.And though not the real reason for prefering a clustered index on each table, a nice added benefit is that you can henceforth reclaim lost space by simply rebuilding the index. With Enterprise Edition, that can even be done without downtime, by using the online rebuild feature!

RE: Reclaiming freed space

SQLRNNR — Tue, 25 Aug 2009 11:41:18 GMT

This was a good question. I would have initially ticked the truncate table option but ran some testing and research first. Though accurate, I am not sure I agree with dropping the clustered index - I think I would rather drop the nonclustered index and leave a clustered index on the table.Thanks for the good question.

RE: Reclaiming freed space

Hugo Kornelis — Thu, 06 Aug 2009 02:27:46 GMT

[quote][b]Chad Crawford (8/6/2009)[/b][hr][quote]Wow, that is interesting. I wonder what it is underneath that makes the difference. I'm cleaning up a lot of old data (horizontal and vertical), so this topic is quite opportune.[quote]However, this is a bit off-topic here, as the QotD was not about reclaiming free space after deleting [b]all or any rows[/b] from a table, but after deleting [b]one column[/b] without removing any row.[/quote]Quite right. I'll take it elsewhere if I decide to pursue it.[/quote]Well, you know about the vertical cleanup by know. :-)For the details of horizontal celanup, I'd suggest posting a question to the newsgroup microsoft.public.sqlserver.server (if you have newsgroup access). Kalen Delaney regularly reads and replies there, and if anyone knows the fine details, she does!

RE: Reclaiming freed space

Chad Crawford — Thu, 06 Aug 2009 01:03:03 GMT

[quote]I'm not sure if Chad's "brutal" comment was intended as a compliment, but I'll take it as such[/quote]Whoops - I didn't even think that might be taken as a complaint, it certainly wasn't meant that way. I was using it in the same way you might call a long, hard workout "brutal". It's good for you, you're glad it's over and you feel like you have conquered something. Thanks for assuming positive intent - there was definitely nothing negative intended there.[quote]This is strange. I ran some tests as well, and I found that deleting all rows does free up most space, but not all. I also found that most space is released immediately, but there's also a bit released only after a second or so. And the most surprising is that the amount of unreleased space seems to depend on various factors I could not put my finger on - at one time, I consistently was left with about 200 KB or so after deleting (with a 10,000 row test); later when I tried again (but apparently not with the exact same table and data), I was consistently left with a mere 32KB.[/quote]Wow, that is interesting. I wonder what it is underneath that makes the difference. I'm cleaning up a lot of old data (horizontal and vertical), so this topic is quite opportune.[quote]However, this is a bit off-topic here, as the QotD was not about reclaiming free space after deleting [b]all or any rows[/b] from a table, but after deleting [b]one column[/b] without removing any row.[/quote]Quite right. I'll take it elsewhere if I decide to pursue it. Thanks Hugo, and welcome back.Chad

RE: Reclaiming freed space

Hugo Kornelis — Wed, 05 Aug 2009 10:05:41 GMT

[quote][b]Cliff Jones (8/5/2009)[/b][hr][quote][b]Hugo Kornelis (8/5/2009)[/b][hr]You'll see that after the TRUNCATE TABLE, the space used is zero - but after the INSERT INTO, it is right back at its original size, even though Column2 has already been dropped. Only after creating and dropping a clustered index (or creating, copying, and renaming a new table) will the data size of the table be reduced.[/quote]That's very interesting. I would not have guessed that it would work that way. Thanks for the clear and concise explanation.[/quote]My pleasure! ;-)

RE: Reclaiming freed space

Cliff Jones — Wed, 05 Aug 2009 09:39:12 GMT

[quote][b]Hugo Kornelis (8/5/2009)[/b][hr]You'll see that after the TRUNCATE TABLE, the space used is zero - but after the INSERT INTO, it is right back at its original size, even though Column2 has already been dropped. Only after creating and dropping a clustered index (or creating, copying, and renaming a new table) will the data size of the table be reduced.[/quote]That's very interesting. I would not have guessed that it would work that way. Thanks for the clear and concise explanation.

RE: Reclaiming freed space

Hugo Kornelis — Wed, 05 Aug 2009 08:57:05 GMT

[quote][b]stevecable (7/31/2009)[/b][hr]The answer claims "Even though TRUNCATE TABLE does remove all the data pages and could theoretically be a great time to change the metadata, it doesn't work this way (as can easily be verified by testing). "So, I tested truncation, and it sure seems to me like all the space is reclaimed. I get zero's across the board on data and indexes. I have to admit being baffled why truncation is not a correct option for reclaiming space. Hmmm.[/quote]Hi Steve,Yes, truncating a table will free up all space allocated to it. No problem. But that is not the scenario this question was about. You have to relaod the rows that were there (without the now unused Column2). If you do that and check sp_spaceused, you'll see that the 200 bytes per row previously required for Column2 are still used for each row - in other words, after truncating and reloading the data, you still have not reclaimed any space. Run the sample below (make sure to create and populate MyTable2 first) to check:[code="sql"]DROP TABLE MyTable;goCREATE TABLE MyTable (Column1 int NOT NULL, Column2 char(200) NOT NULL, Column3 varchar(40) NOT NULL, PRIMARY KEY NONCLUSTERED(Column1));INSERT INTO MyTableSELECT * FROM MyTable2goEXEC sp_spaceused MyTable;ALTER TABLE MyTableDROP COLUMN Column2;EXEC sp_spaceused MyTable;TRUNCATE TABLE MyTable;EXEC sp_spaceused MyTable;INSERT INTO MyTable(Column1, Column3)SELECT Column1, Column3FROM MyTable2;EXEC sp_spaceused MyTable;CREATE CLUSTERED INDEX xx ON MyTable(Column1)DROP INDEX MyTable.xxEXEC sp_spaceused MyTable;go[/code]You'll see that after the TRUNCATE TABLE, the space used is zero - but after the INSERT INTO, it is right back at its original size, even though Column2 has already been dropped. Only after creating and dropping a clustered index (or creating, copying, and renaming a new table) will the data size of the table be reduced.

RE: Reclaiming freed space

Hugo Kornelis — Wed, 05 Aug 2009 08:50:51 GMT

[quote][b]sean redmond (7/28/2009)[/b][hr]Wouldn't shrinking the DB also reclaim the space lost by the dropped column?[/quote]Frankly, I have no idea. I guess I could test but I can't really be bothered. I prefer not to shrink me databases at all, because shrinking comes with a bucketload of undesired side effects.See [url=http://www.karaszi.com/SQLServer/info_dont_shrink.asp]http://www.karaszi.com/SQLServer/info_dont_shrink.asp[/url] for the gory details.

RE: Reclaiming freed space

Hugo Kornelis — Wed, 05 Aug 2009 08:48:59 GMT

[quote][b]angelo.cas (7/28/2009)[/b][hr]Well I would have added and removed the clustered index. It's by far the simplest process that would have less room for fault or down time on the table.Why you would do all the other stuff is beyond me, just asking for trouble.[/quote]Hi Angelo,Adding and removing a clustered index is the simplest, that is true. That does not automatically make it the best, and it will definitely not yield the lowest amount of down time automatically.Remember that nonclustered indexes include either a reference to the clustered index if there is one, or a reference to the database page otherwise. Adding a clustered index forces each nonclustered index (one in the case of this example) to be rebuilt. And that goes for removing a clustered index as well.Let's compare two approaches: the copy/drop/rename approach and the create/drop clustered index approach.[u]Create/drop clustered index[/u]Creating the clustered index involves the following steps:* Sort data (one million rows!)* Copy data to new data pages* Free previously allocated data pages* Rebuild one nonclustered indexDropping the clustered index is cheaper:* Drop the index (metadata only)* Rebuild one nonclustered index[u]Copy/drop/rename table[/u]Copying the data to a new table:* Copy data to new data pages* Build one nonclustered indexDrop old table:* Drop table and index (metadata only)* Free previously allocated data pages (for index as well)Rename table:* Metadata onlyAs you see, adding and removing a clustered index causes the nonclustered index to be rebuilt twice, whereas the copy/drop/rename scenarion has to build the new nonclustered only once. This can add up significantly if the table has several nonclustered indexes. Other than that, the create/drop clustered index method also incurs a sort operation that is not really required.I think the "best" option depends. How afraid are you of data loss due to operator error? How many nonclustered indexes are involved? How long is the time window available for the operation? If you need 24x7 availability and you have enterprise edition, you can use its online index rebuilding capabilities (though I'd have to check if they work for creating and dropping a clustered index as well) Etc, etc, etc.

RE: Reclaiming freed space

Hugo Kornelis — Wed, 05 Aug 2009 08:34:17 GMT

[quote][b]Stuart M. Garner (7/28/2009)[/b][hr]Interesting that in the explanation for the "correct" answer it states:Creating a clustered index does involve a rebuild of all the data pages, so this will also change the metadata and free up the space previously taken by Column2. Stated in the referenced article athttp://msdn.microsoft.com/en-us/library/ms177563.aspxit states:Note: Dropping a column does not reclaim the disk space of the column. You may have to reclaim the disk space of a dropped column when the row size of a table is near, or has exceeded, its limit. Reclaim space by creating a clustered index on the table or rebuilding an existing clustered index by using ALTER INDEX [ http://msdn.microsoft.com/en-us/library/ms188388.aspx ] . As observed by others, it appears there is more than one way to do this.:-)[/quote]Hi Stuart,Yes, there is more than one way to do this. I even included three ways, and you had to tick alll of them to score a point :-PI have a hunch that you think the text you quote contradicts what I wrote, but I fail to see the contradiction. Or am I misunderstanding the point of your post?EDIT: Just saw that the quote includes using ALTER INDEX to rebuild a clustered index. Yes, that would work too - if there is an existing clustered index to rebuild. In this case, there was only a nonclustered index, so rebuilding was not an option.

RE: Reclaiming freed space

Hugo Kornelis — Wed, 05 Aug 2009 08:32:27 GMT

[quote][b]andrew.me.ph (7/28/2009)[/b][hr]As to me, this is a silly answer: since both building clustered index and copying data multiple times solve "the problem" - presenting multiple copy as a correct answer is really silly. It is like saying that doing more operations (involving both more scripting amd more database operations) is better than doing less scripting and less database operations that lead to the same result. I am really surprised with the answer. I wonder who has decided which answer is the correct one.[/quote]Hi Andrew,I decided which answers are correct. :-DI agree that there is no need to perform all the actions in the correct answers. One is enough. Pick whichever you like best. :-)I disagree that this makes the answer silly. The question specifically asked which of the presented strategies "can be used", not which one "should be used". (And in one of the later messages, you'll also see that there is no clear-cut reply to that question). So the question was designed to test if you know which actions will free up space after dropping a column and which won't, not to ask for a "best practice".EDIT: Fixed the smileys

RE: Reclaiming freed space

Hugo Kornelis — Wed, 05 Aug 2009 08:28:54 GMT

[quote][b]mtassin (7/28/2009)[/b][hr]The Garbage collection process is also a correct answer.[quote]][b]SQL Server 2005 Books Online April 2006 Help File[/b][hr][i](snip - HK)[/i][/quote]So sp_spaceused for a large table (1 million rows seems large enough to me) will not report the space freed up until the batches of physical phases are complete. Unless sp_spaceused @objectusage='updateusage' is used.Calling it garbage collection might be a stretch, but it is serving that purpose behind the scenes.[/quote]I have no objection at all to calling this garbage collection. But that does not make it a correct answer. The Books Online fragment you quote specifically states that this process kicks in "When you drop or rebuild large indexes, or drop or truncate large tables". In the question's scenario, a column was dropped and the garbage collection answer suggested that the table's pages will be automatically rebuilt to free up the associated disk space by some automatic process. This is not true. If you drop a column from a large table without doing anything else, the space taken by that column will continue to be reserved forever, unless you create a new table or create, drop, or rebuild a clustered index.

RE: Reclaiming freed space

Hugo Kornelis — Wed, 05 Aug 2009 08:25:12 GMT

[quote][b]Chad Crawford (7/28/2009)[/b][hr]Hugo (et al.), I was running some scripts (in 2K5) based on this question to learn a little more and I noticed that the delete doesn't seem to return any space at all, even before reloading all the data back in?!? I filled your table with 1,000,000 records, copied the rows to a new table, then did a delete with no where (scary all by itself) and sp_spaceused showed the same amount of reserved space as before I ran the delete. Is it really holding on to all those pages still? I looked in the BOL under [url=http://msdn.microsoft.com/en-us/library/ms177495.aspx]Dropping and Rebuilding Large Objects[/url] and it seems to indicate that the cleanup doesn't happen until after the transaction commits, but the result is the same whether or not I wrap it in a transaction. It's the oddest thing - sp_spaceused shows 0 rows taking up 47MB. :ermm: I understand now why the same space is used after reloading the data, but what about when there is no data at all? I never expected that. Thanks,Chad[/quote]Hi Chad,This is strange. I ran some tests as well, and I found that deleting all rows does free up most space, but not all. I also found that most space is released immediately, but there's also a bit released only after a second or so. And the most surprising is that the amount of unreleased space seems to depend on various factors I could not put my finger on - at one time, I consistently was left with about 200 KB or so after deleting (with a 10,000 row test); later when I tried again (but apparently not with the exact same table and data), I was consistently left with a mere 32KB.However, this is a bit off-topic here, as the QotD was not about reclaiming free space after deleting [b]all or any rows[/b] from a table, but after deleting [b]one column[/b] without removing any row.

RE: Reclaiming freed space

Hugo Kornelis — Wed, 05 Aug 2009 08:20:38 GMT

[quote][b]nico van niekerk (7/28/2009)[/b][hr]I clicked the correct answer but it still said that I was wrong and cited an option I didn't click. Then, it is not "than", it is "then". It's a pity that the effort wasn't debugged.[/quote]Hi Nico,I'm not sure what to make of your first sentence. Do you mean that you disagree with the options I marked as correct? Or do you mean that you site malfunctioned, claiming you clicked an option you didn't actually click? If the latter, you'll have to take it up with Steve, I can't help you there. And in the former case, I can only reallly comment if you tell me WHICH of the options you disagree with.Your comment about my spelling is correct. I could try to hide behind me not being a native speaker, but in all honesty I can only admit that this is an error I should be able to avoid. Sorry for not proofreading the question a bit better. I hope this did't cause you to misunderstand the question.I know Steve does look at submitted QotD entries before posting them, but he can never catch all errors. In the case of a simple then/than error, I think he can be forgiven.

RE: Reclaiming freed space

Hugo Kornelis — Wed, 05 Aug 2009 08:15:05 GMT

Hi all,First: Apologies for the late replies. Unfortunately, Steve ran this question just during my holiday, so I was busy soaking in sun, blissfully unaware of your reactions. I am only now clearing out my backlog.Several people have complimented me on the question. To you, I say: Thank you! Good to get the positive feedback.I'm not sure if Chad's "brutal" comment was intended as a compliment, but I'll take it as such: I don't submit many questions, but when I do I want to try to ask a question that teaches many readers something they didn't know, which automatically translates to a tough question. If 95% get it right, I failed because only 5% learned from it. If 20% get it right due to a badly worded question, I failed because I try think the QotD should test SQL Server skills, not reading or interpretation skills. But if 20% get it right because the other 80% didn't know, than I succeeded in teaching those 80% something new.As to RBarryYoung's comment, I agree. The submitter of the question does not get to set the point total, that's up to Steve - and I too am surprised to see this being rated as a one-pointer.Several people have also posted constructive criticism. Thanks to them as well - as good and nice as positive feedback is, nothing is better for increasin my own knowledge than cionctructive criticism. I will address each of those messages in individual replies after this one.

RE: Reclaiming freed space

Cliff Jones — Fri, 31 Jul 2009 14:47:04 GMT

Interesting. I took Hugo's word for it and didn't bother to test it.

RE: Reclaiming freed space

stevecable — Fri, 31 Jul 2009 14:26:43 GMT

The answer claims "Even though TRUNCATE TABLE does remove all the data pages and could theoretically be a great time to change the metadata, it doesn't work this way (as can easily be verified by testing). "So, I tested truncation, and it sure seems to me like all the space is reclaimed. I get zero's across the board on data and indexes. I have to admit being baffled why truncation is not a correct option for reclaiming space. Hmmm.

RE: Reclaiming freed space

RBarryYoung — Wed, 29 Jul 2009 15:56:21 GMT

Heh, missed it.Steve: This seems like an awfully difficult question for only one point. In fact, the Check-Box questions are inherently more difficult than the Radio-Button questions, maybe they should always start at 2 points?

RE: Reclaiming freed space

Rob Goddard — Wed, 29 Jul 2009 07:35:43 GMT

Noooooooo! I got it wrong and I was doing sooo well! :-DExcellent question, first QotD in a long while that actually made me shift into a sitting up position in my chair.. :hehe:

RE: Reclaiming freed space

Sean Redmond — Tue, 28 Jul 2009 23:35:48 GMT

Wouldn't shrinking the DB also reclaim the space lost by the dropped column?

RE: Reclaiming freed space

angelo.cas — Tue, 28 Jul 2009 15:59:30 GMT

Well I would have added and removed the clustered index. It's by far the simplest process that would have less room for fault or down time on the table.Why you would do all the other stuff is beyond me, just asking for trouble.

RE: Reclaiming freed space

SMGarner — Tue, 28 Jul 2009 13:41:00 GMT

Interesting that in the explanation for the "correct" answer it states:Creating a clustered index does involve a rebuild of all the data pages, so this will also change the metadata and free up the space previously taken by Column2. Stated in the referenced article athttp://msdn.microsoft.com/en-us/library/ms177563.aspxit states:Note: Dropping a column does not reclaim the disk space of the column. You may have to reclaim the disk space of a dropped column when the row size of a table is near, or has exceeded, its limit. Reclaim space by creating a clustered index on the table or rebuilding an existing clustered index by using ALTER INDEX [ http://msdn.microsoft.com/en-us/library/ms188388.aspx ] . As observed by others, it appears there is more than one way to do this.:-)

RE: Reclaiming freed space

andrew.me.ph — Tue, 28 Jul 2009 13:10:17 GMT

As to me, this is a silly answer: since both building clustered index and copying data multiple times solve "the problem" - presenting multiple copy as a correct answer is really silly. It is like saying that doing more operations (involving both more scripting amd more database operations) is better than doing less scripting and less database operations that lead to the same result. I am really surprised with the answer. I wonder who has decided which answer is the correct one.

RE: Reclaiming freed space

mtassin — Tue, 28 Jul 2009 09:35:44 GMT

The Garbage collection process is also a correct answer.[quote]][b]SQL Server 2005 Books Online April 2006 Help File[/b][hr]When you drop or rebuild large indexes, or drop or truncate large tables, the SQL Server 2005 Database Engine defers the actual page deallocations, and their associated locks, until after a transaction commits. This implementation supports both autocommit and explicit transactions in a multiuser environment, and applies to large tables and indexes that use more than 128 extents. The Database Engine avoids the allocation locks that are required to drop large objects by splitting the process in two separate phases: logical and physical. In the logical phase, the existing allocation units used by the table or index are marked for deallocation and locked until the transaction commits. With a clustered index that is dropped, the data rows are copied and then moved to new allocation units created to the store either a rebuilt clustered index, or a heap. (In the case of an index rebuild, the data rows are sorted also.) When there is a rollback, only this logical phase needs to be rolled back. The physical phase occurs after the transaction commits. The allocation units marked for deallocation are physically dropped in batches. These drops are handled inside short transactions that occur in the background, and do not require lots of locks.Because the physical phase occurs after a transaction commits, the storage space of the table or index might still appear as unavailable. If this space is required for the database to grow before the physical phase is completed, the Database Engine tries to recover space from allocation units marked for deallocation. To find the space currently used by these allocation units, use the sys.allocation_units catalog view.[/quote]So sp_spaceused for a large table (1 million rows seems large enough to me) will not report the space freed up until the batches of physical phases are complete. Unless sp_spaceused @objectusage='updateusage' is used.Calling it garbage collection might be a stretch, but it is serving that purpose behind the scenes.

RE: Reclaiming freed space

OCTom — Tue, 28 Jul 2009 08:52:50 GMT

Great question. This was a real brain tickler for me. I like the first answer. Just wait a while. If it were only that easy. :-D

RE: Reclaiming freed space

Chad Crawford — Tue, 28 Jul 2009 08:51:03 GMT

Hugo (et al.), I was running some scripts (in 2K5) based on this question to learn a little more and I noticed that the delete doesn't seem to return any space at all, even before reloading all the data back in?!? I filled your table with 1,000,000 records, copied the rows to a new table, then did a delete with no where (scary all by itself) and sp_spaceused showed the same amount of reserved space as before I ran the delete. Is it really holding on to all those pages still? I looked in the BOL under [url=http://msdn.microsoft.com/en-us/library/ms177495.aspx]Dropping and Rebuilding Large Objects[/url] and it seems to indicate that the cleanup doesn't happen until after the transaction commits, but the result is the same whether or not I wrap it in a transaction. It's the oddest thing - sp_spaceused shows 0 rows taking up 47MB. :ermm: I understand now why the same space is used after reloading the data, but what about when there is no data at all? I never expected that. Thanks,Chad

RE: Reclaiming freed space

Chad Crawford — Tue, 28 Jul 2009 07:52:32 GMT

Drat, lost my streak! Brutal question with so many options, but I did learn something new about internals! Thanks Hugo.

RE: Reclaiming freed space

nico van niekerk — Tue, 28 Jul 2009 07:48:44 GMT

I clicked the correct answer but it still said that I was wrong and cited an option I didn't click. Then, it is not "than", it is "then". It's a pity that the effort wasn't debugged.

RE: Reclaiming freed space

Cliff Jones — Tue, 28 Jul 2009 07:25:46 GMT

I like most people thought that truncating the table should work also. Good question, thanks.

RE: Reclaiming freed space

dave.farmer — Tue, 28 Jul 2009 06:35:23 GMT

What an excellent and highly educational question - not often does a QotD make me rethink established strategies.

Reclaiming freed space

Hugo Kornelis — Tue, 28 Jul 2009 00:26:20 GMT

Comments posted to this topic are about the item [B]Reclaiming freed space[/B]