Growing tropical fruits in Seattle without a proper greenhouse is a bit of a pain. They tend to prefer much more acidic soil and high humidity.
However I've had a bit of success with a few of my plants. I've made it to the next step of acclimating my 2 longan plants to our Seattle summer weather. They've been living in a very humid indoor greenhouse and it's put on about 6 inches of growth in a couple of months.
The keys to success have been:
Low PH Soil 5.5 or lower + High Humidity (60+) + Low PH water + Light and long days = Great growth.
I've been lowering my PH with Azalea fertilizer, Sulfur, and Coffee grinds. I initially used peat but it attracts way too many fungus gnats.
Try not to use tap water to water your plants with. It's a quick way to kill them.
In the colder months when you have to keep them inside, use a clear bin or plastic greenhouse to keep the humidity up. Buy a $9 humidity gauge. It's worth every penny.
I don't think I can expect fruits from these trees. It's more of a test to keep them alive here.