This is the case and solution to my server a SQL Server 2008R2 installation.
Cause of pain, the job agent history log was let unlimited, none of the history ticks were ticked.
I was fighting on two fronts, small HDD size and large MSDB and TEMPDB system databases.
1. checked MSDB size, Data 7gb Log 33gb - this was due to running the sp_purge_jobhistory - bloats MSDB and tempDB
2. checked via script the msdb for large objects; table dbo.sysjobhistory had loads of rows and size wise was the bulk of the 7gb MSDB.
3. i made sure i can shrink tempdb (restart sqls erver) and MSDB by putting it to single mode, shrink and multi again.
4. Kept run sp_purge_jobhistory with shorter and shorter 'oldest date' keeping an eye on size of MSDB and TEMPDB shrinking as i go. this was probably not essential but i am a bit careful with large purges. you could try one off, keep only 14 days for example.
5. rerun the object dbo.sysjobhistor size check in MSDB, noticed that the actual size was due to the table index being huge although the rows are not only 100.
6. open mdsb table dbo.sysjobhistory and found the two index the clustered and the non clustered; right click and check fragmentation. 96% - ticked Reorganise.
7. right click on MSDB and task shring database - this shrunk the msdb to a mnimum size 120 MB
Went on Sql Agent job and clicked for the history - no time out and faster than ever!
MSDB Object size code.
, obj = SCHEMA_NAME(o.[schema_id]) + '.' + o.name
FROM sys.objects o
, total_size = CAST(SUM(a.total_pages) * 8. / 1024 AS DECIMAL(18,2))
, total_rows = SUM(CASE WHEN i.index_id IN (0, 1) AND a.[type] = 1 THEN p.[rows] END)
FROM sys.indexes i
JOIN sys.partitions p ON i.[object_id] = p.[object_id] AND i.index_id = p.index_id
JOIN sys.allocation_units a ON p.[partition_id] = a.container_id
WHERE i.is_disabled = 0
AND i.is_hypothetical = 0
GROUP BY i.[object_id]
) i ON o.[object_id] = i.[object_id]
WHERE o.[type] IN ('V', 'U', 'S')
ORDER BY i.total_size DESC