Extremely poor concurrency #8

Author: @ianthetechie

Hey guys,

First off, I want to say thanks for this fantastic work. This is a fantastic little tool that will make vector tile serving much more accessible.

However, as it stands right now, a single sqlalchemy session is used, which creates a very nasty bottleneck. If you try to put it under any significant load, you'll quickly notice that your machine never uses more than 1 core and load average is capped at 1.0. If nobody else is working on this, I could probably submit a PR to clean this up in a week or two.

Author: @ianthetechie Hey guys, First off, I want to say thanks for this fantastic work. This is a fantastic little tool that will make vector tile serving much more accessible. However, as it stands right now, a single sqlalchemy session is used, which creates a very nasty bottleneck. If you try to put it under any significant load, you'll quickly notice that your machine never uses more than 1 core and load average is capped at 1.0. If nobody else is working on this, I could probably submit a PR to clean this up in a week or two.

Author: @boldtrn

Any updates on this @ianthetechie. I have seen you mention to work on this a couple of times.

Author: @boldtrn Any updates on this @ianthetechie. I have seen you mention to work on this a couple of times.

Author: @ianthetechie

I haven't worked on it super actively lately, but the short summary is above.

The slightly longer version is that rather than reusing the session each time, you need to fire off a call to engine.connect() every time inside get_mvt. It's slightly more complicated than just doing this though, because you also need to set up a hook for preparing the SQL statement etc. I'm not 100% convinced this is the proper way to do it yet, but here's a gist if you're interested: https://gist.github.com/ianthetechie/2aefcd3687e8d539e4b891f92b2c642e

Finally, take note of issues #6 and #5. I think #5 will be fixed upstream in the upcoming postgis release. #6 is trickier. There is a WIP PR open for this right now, but it's a bit of a wreck from my looking through it. I've had relatively bad luck trying to fix it myself. While this isn't typically going to be an issue with vector tiles that you're just using for vector use, having the buffers in place for rasterizing really helps if you happen to be doing that. My attempts so far though have just yielded really large tiles that still had major issues when put through a rasterizer like tileserver GL (and ironically were missing some data).

Hope this helps! Cheers!

Author: @ianthetechie I haven't worked on it super actively lately, but the short summary is above. The slightly longer version is that rather than reusing the session each time, you need to fire off a call to `engine.connect()` every time inside `get_mvt`. It's slightly more complicated than just doing this though, because you also need to set up a hook for preparing the SQL statement etc. I'm not 100% convinced this is the proper way to do it yet, but here's a gist if you're interested: https://gist.github.com/ianthetechie/2aefcd3687e8d539e4b891f92b2c642e Finally, take note of issues #6 and #5. I think #5 will be fixed upstream in the upcoming postgis release. #6 is trickier. There is a WIP PR open for this right now, but it's a bit of a wreck from my looking through it. I've had relatively bad luck trying to fix it myself. While this isn't typically going to be an issue with vector tiles that you're *just* using for vector use, having the buffers in place for rasterizing really helps if you happen to be doing that. My attempts so far though have just yielded really large tiles that still had major issues when put through a rasterizer like tileserver GL (and ironically were missing some data). Hope this helps! Cheers!

Author: @boldtrn

Thanks for the detailed explanation and hints @ianthetechie. This helps a lot.

Author: @boldtrn Thanks for the detailed explanation and hints @ianthetechie. This helps a lot.

Labels Milestones

Extremely poor concurrency #8