AFAIK all the strategies to speed up reading Excel involve finding ways to avoid using the COM model.

You can cause excel to save the data as .cvs and then read the .cvs file.

If the spreadsheet is laid out as a nice table, you can use ODBC to read the file.

But I don't know of a way to get at the color data using these methods.

If you control the layout of the spreadsheets, you could write a Function in VBA in the spreadsheet. Said function would return a number indicating what color was found in the referenced cell. Insert a "helper" column next to the column you are interested in. Refer to the function in the helper column - then when you read the file with ODBC or save it as .cvs, you will have a number value you can inspect.

An example is here:

http://stackoverflow.com/questions/5110963/excel-get-cell-color