Impact on database

Hi Bram,

Great component so far. I'm a bit testing with it. If I decide to keep using it I will certainly donate something.

I'm wondering: it is possible to import hunderds or thousands of feed-items. Do you have any experience what this will do with the speed of the website and database?

Any advise?

Thanks,
Jeroen

depends a lot on your server configuration and the amount of traffic. More traffic might actually result in a overall better performance due to query caching.


I have a few notes on the topic : affiliatefeeds.nl/articles/11-feed-impor...formance-issues.html

a few thousand items should not be a problem on any server.

MySQL tends to get slower as the number of items grow. You can do a bit of tuning like key-write-delay and allocating more memory for caching queries and indexes helps quite a bit. A limit is somewhere at 250K. Then it get's slower again depending on your CPU/Memory/mysql-tuning. (And if you put a 250K database on a cheap shared host you site will be suspended before your first sale is approved)

I have a site with 650K items, that didn't perform to well using mysql,

On this site I'm now using Elasticsearch as engine. This Lucene based database does a better job searching normalized data like the datafeeds. On the other side Elasticsearch is relative slow in indexing and especially updating data.

above post applies to the front end.

another problem might ( will ) occur when importing large feeds.

While the parser itself is perfectly capable of importing any sized feeds the web based backend is not. You will run into php and http timeouts.

To import large feeds you might/will need to run the import using the command line.