You are reinventing wheel sieving

Faster sieves go one step further: you don't need to store multiples of 2 (which you already know), 3, 5 and so on. For instance, if you remove multiples of 2 and 3, you only need to take care of numbers that have the form 6n+1 or 6n+5. This means that in a single byte you can store 6x4 candidates.