skylarmt
/
postserve
1
0
Fork 0
Code Issues 6 Pull Requests 5 Releases Wiki Activity

Labels Milestones

Improve concurrency #14

Manually merged
skylarmt merged 1 commits from eyusupov/concurrency into master 5 years ago
Conversation 0 Commits 1 Files Changed 3
skylarmt commented 5 years ago
Owner

Author: @eyusupov

  • Update Python to use async/await
  • Use asyncpg
  • Use connection pool

I didn't have a powerful server to test it (on my machine the bottleneck was the query execution time), but it does seem to improve concurrency:

[I 191021 01:16:39 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4852834.051769271, 7592337.145509987), ST_Point(4891969.810251279, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:16:39 server:97] calculated response
[I 191021 01:16:39 web:1971] 200 GET /tiles/10/636/317.pbf (172.19.0.1) 200.67ms
[I 191021 01:16:39 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4852834.051769271, 7631472.903991994), ST_Point(4891969.810251279, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:16:39 server:97] calculated response
[I 191021 01:16:39 web:1971] 304 GET /tiles/10/636/316.pbf (172.19.0.1) 120.94ms
[I 191021 01:16:39 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4891969.810251279, 7631472.903991994), ST_Point(4931105.568733291, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:16:39 server:97] calculated response
[I 191021 01:16:39 web:1971] 304 GET /tiles/10/637/316.pbf (172.19.0.1) 87.38ms
[I 191021 01:16:39 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4813698.293287259, 7592337.145509987), ST_Point(4852834.051769271, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:16:39 server:97] calculated response
[I 191021 01:16:39 web:1971] 304 GET /tiles/10/635/317.pbf (172.19.0.1) 83.32ms
[I 191021 01:16:39 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4813698.293287259, 7631472.903991994), ST_Point(4852834.051769271, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:16:40 server:97] calculated response
[I 191021 01:16:40 web:1971] 200 GET /tiles/10/635/316.pbf (172.19.0.1) 882.31ms
[I 191021 01:16:40 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4891969.810251279, 7592337.145509987), ST_Point(4931105.568733291, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:16:51 server:97] calculated response
[I 191021 01:16:51 web:1971] 200 GET /tiles/10/637/317.pbf (172.19.0.1) 10923.63ms
[I 191021 01:16:51 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7631472.903991994), ST_Point(4970241.327215302, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:16:51 server:97] calculated response
[I 191021 01:16:51 web:1971] 200 GET /tiles/10/638/316.pbf (172.19.0.1) 235.96ms
[I 191021 01:16:51 server:95] received request
EXECUTE gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7592337.145509987), ST_Point(4970241.327215302, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:16:51 server:97] calculated response

After:

[I 191021 01:26:23 server:86] received request
[I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4852834.05176927, 7592337.145509987), ST_Point(4891969.810251281, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:26:23 server:86] received request
[I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4891969.810251281, 7592337.145509987), ST_Point(4931105.568733291, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:26:23 server:86] received request
[I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4852834.05176927, 7631472.903991994), ST_Point(4891969.810251281, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:26:23 server:86] received request
[I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4891969.810251281, 7631472.903991994), ST_Point(4931105.568733291, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:26:23 server:86] received request
[I 191021 01:26:24 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4813698.2932872595, 7592337.145509987), ST_Point(4852834.05176927, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:26:24 server:86] received request
[I 191021 01:26:24 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4813698.2932872595, 7631472.903991994), ST_Point(4852834.05176927, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:26:27 server:89] calculated response
[I 191021 01:26:28 server:89] calculated response
[I 191021 01:26:28 server:89] calculated response
[I 191021 01:26:28 web:2246] 200 GET /tiles/10/635/316.pbf (172.19.0.1) 4864.85ms
[I 191021 01:26:28 web:2246] 200 GET /tiles/10/637/316.pbf (172.19.0.1) 4869.97ms
[I 191021 01:26:28 web:2246] 200 GET /tiles/10/636/316.pbf (172.19.0.1) 4873.18ms
[I 191021 01:26:28 server:86] received request
[I 191021 01:26:28 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7592337.145509987), ST_Point(4970241.327215301, 7631472.903991994)), 545978.7734655447, 256, 256);
[I 191021 01:26:28 server:86] received request
[I 191021 01:26:28 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7631472.903991994), ST_Point(4970241.327215301, 7670608.662474007)), 545978.7734655447, 256, 256);
[I 191021 01:26:29 server:89] calculated response
[I 191021 01:26:29 web:2246] 200 GET /tiles/10/635/317.pbf (172.19.0.1) 5939.82ms
[I 191021 01:26:29 server:89] calculated response
[I 191021 01:26:29 web:2246] 304 GET /tiles/10/638/316.pbf (172.19.0.1) 971.60ms
[I 191021 01:26:29 server:89] calculated response
[I 191021 01:26:29 web:2246] 304 GET /tiles/10/638/317.pbf (172.19.0.1) 1254.28ms
[I 191021 01:26:37 server:89] calculated response
[I 191021 01:26:37 web:2246] 200 GET /tiles/10/636/317.pbf (172.19.0.1) 14030.48ms
[I 191021 01:26:38 server:89] calculated response
[I 191021 01:26:38 web:2246] 200 GET /tiles/10/637/317.pbf (172.19.0.1) 14568.89ms

Note hat individual SQL query times have increased - I'm not sure if it's because now several queries have to be processed at once by my underpowered PG server or something else.
Would be great if somebody else could check that. Postserve itself seems to be more concurrent now.

In the future, function created by generate-sqltomvt could be used instead of the built in one.

Author: @eyusupov - Update Python to use async/await - Use asyncpg - Use connection pool I didn't have a powerful server to test it (on my machine the bottleneck was the query execution time), but it does seem to improve concurrency: ``` [I 191021 01:16:39 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4852834.051769271, 7592337.145509987), ST_Point(4891969.810251279, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:16:39 server:97] calculated response [I 191021 01:16:39 web:1971] 200 GET /tiles/10/636/317.pbf (172.19.0.1) 200.67ms [I 191021 01:16:39 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4852834.051769271, 7631472.903991994), ST_Point(4891969.810251279, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:16:39 server:97] calculated response [I 191021 01:16:39 web:1971] 304 GET /tiles/10/636/316.pbf (172.19.0.1) 120.94ms [I 191021 01:16:39 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4891969.810251279, 7631472.903991994), ST_Point(4931105.568733291, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:16:39 server:97] calculated response [I 191021 01:16:39 web:1971] 304 GET /tiles/10/637/316.pbf (172.19.0.1) 87.38ms [I 191021 01:16:39 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4813698.293287259, 7592337.145509987), ST_Point(4852834.051769271, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:16:39 server:97] calculated response [I 191021 01:16:39 web:1971] 304 GET /tiles/10/635/317.pbf (172.19.0.1) 83.32ms [I 191021 01:16:39 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4813698.293287259, 7631472.903991994), ST_Point(4852834.051769271, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:16:40 server:97] calculated response [I 191021 01:16:40 web:1971] 200 GET /tiles/10/635/316.pbf (172.19.0.1) 882.31ms [I 191021 01:16:40 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4891969.810251279, 7592337.145509987), ST_Point(4931105.568733291, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:16:51 server:97] calculated response [I 191021 01:16:51 web:1971] 200 GET /tiles/10/637/317.pbf (172.19.0.1) 10923.63ms [I 191021 01:16:51 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7631472.903991994), ST_Point(4970241.327215302, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:16:51 server:97] calculated response [I 191021 01:16:51 web:1971] 200 GET /tiles/10/638/316.pbf (172.19.0.1) 235.96ms [I 191021 01:16:51 server:95] received request EXECUTE gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7592337.145509987), ST_Point(4970241.327215302, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:16:51 server:97] calculated response ``` After: ``` [I 191021 01:26:23 server:86] received request [I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4852834.05176927, 7592337.145509987), ST_Point(4891969.810251281, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:26:23 server:86] received request [I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4891969.810251281, 7592337.145509987), ST_Point(4931105.568733291, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:26:23 server:86] received request [I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4852834.05176927, 7631472.903991994), ST_Point(4891969.810251281, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:26:23 server:86] received request [I 191021 01:26:23 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4891969.810251281, 7631472.903991994), ST_Point(4931105.568733291, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:26:23 server:86] received request [I 191021 01:26:24 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4813698.2932872595, 7592337.145509987), ST_Point(4852834.05176927, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:26:24 server:86] received request [I 191021 01:26:24 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4813698.2932872595, 7631472.903991994), ST_Point(4852834.05176927, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:26:27 server:89] calculated response [I 191021 01:26:28 server:89] calculated response [I 191021 01:26:28 server:89] calculated response [I 191021 01:26:28 web:2246] 200 GET /tiles/10/635/316.pbf (172.19.0.1) 4864.85ms [I 191021 01:26:28 web:2246] 200 GET /tiles/10/637/316.pbf (172.19.0.1) 4869.97ms [I 191021 01:26:28 web:2246] 200 GET /tiles/10/636/316.pbf (172.19.0.1) 4873.18ms [I 191021 01:26:28 server:86] received request [I 191021 01:26:28 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7592337.145509987), ST_Point(4970241.327215301, 7631472.903991994)), 545978.7734655447, 256, 256); [I 191021 01:26:28 server:86] received request [I 191021 01:26:28 server:70] SELECT gettile(ST_MakeBox2D(ST_Point(4931105.568733291, 7631472.903991994), ST_Point(4970241.327215301, 7670608.662474007)), 545978.7734655447, 256, 256); [I 191021 01:26:29 server:89] calculated response [I 191021 01:26:29 web:2246] 200 GET /tiles/10/635/317.pbf (172.19.0.1) 5939.82ms [I 191021 01:26:29 server:89] calculated response [I 191021 01:26:29 web:2246] 304 GET /tiles/10/638/316.pbf (172.19.0.1) 971.60ms [I 191021 01:26:29 server:89] calculated response [I 191021 01:26:29 web:2246] 304 GET /tiles/10/638/317.pbf (172.19.0.1) 1254.28ms [I 191021 01:26:37 server:89] calculated response [I 191021 01:26:37 web:2246] 200 GET /tiles/10/636/317.pbf (172.19.0.1) 14030.48ms [I 191021 01:26:38 server:89] calculated response [I 191021 01:26:38 web:2246] 200 GET /tiles/10/637/317.pbf (172.19.0.1) 14568.89ms ``` Note hat individual SQL query times have increased - I'm not sure if it's because now several queries have to be processed at once by my underpowered PG server or something else. Would be great if somebody else could check that. Postserve itself seems to be more concurrent now. In the future, function created by generate-sqltomvt could be used instead of the built in one.
skylarmt reviewed 5 years ago
skylarmt left a comment
Poster
Owner

.

.
skylarmt closed this pull request 5 years ago
The pull request has been manually merged as a92bf6288c.
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b eyusupov/concurrency master
git pull origin eyusupov/concurrency

Step 2:

Merge the changes and update on Gitea.
git checkout master
git merge --no-ff eyusupov/concurrency
git push origin master
Sign in to join this conversation.
Reviewers
Request review
No reviewers
Labels
Apply labels
Clear labels   
bug
   
duplicate
   
enhancement
   
help wanted
   
invalid
   
question
   
wontfix
No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Set milestone
Clear milestone
No items
No Milestone
Assignees
Assign users
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: skylarmt/postserve#14
Write Preview
Loading…
Cancel
Save
There is no content yet.
Delete Branch 'eyusupov/concurrency'

Deleting a branch is permanent. It CANNOT be undone. Continue?

No
Yes