blender-open-data.git
4 months agoAdd support for responsive DataTables master
Francesco Siddi [Thu, 13 Dec 2018 15:15:44 +0000 (16:15 +0100)]
Add support for responsive DataTables

4 months agoIntroducing table display type production
Francesco Siddi [Thu, 13 Dec 2018 13:49:07 +0000 (14:49 +0100)]
Introducing table display type

A description and example on how to structure queries to be displayed as tables
is available in the docs.

4 months agoCorrect sorting of Dashboard Queries
Francesco Siddi [Tue, 11 Dec 2018 19:25:11 +0000 (20:25 +0100)]
Correct sorting of Dashboard Queries

4 months agoDocs: Initial documentation
Francesco Siddi [Tue, 11 Dec 2018 16:57:39 +0000 (17:57 +0100)]
Docs: Initial documentation

4 months agoUpdate dependencies
Francesco Siddi [Tue, 11 Dec 2018 16:56:15 +0000 (17:56 +0100)]
Update dependencies

4 months agoIntroducing Dashboards
Francesco Siddi [Tue, 11 Dec 2018 16:15:46 +0000 (17:15 +0100)]
Introducing Dashboards

With dashboards it is possible to organize and group Predefined Queries together.
Run ./manage.py loaddata dashboards to load a predefined dashboard.

5 months agoelastic_dump management command: be silent when -v0 is passed
Sybren A. Stüvel [Tue, 20 Nov 2018 15:23:17 +0000 (16:23 +0100)]
elastic_dump management command: be silent when -v0 is passed

The script has been running for months without problem, so I don't want
an email every day about it any more until it sees an error.

5 months agoDelay ElasticSearch connection until necessary
Sybren A. Stüvel [Tue, 20 Nov 2018 15:22:41 +0000 (16:22 +0100)]
Delay ElasticSearch connection until necessary

5 months agoFixed 404 response on non-existant benchmark URL
Sybren A. Stüvel [Tue, 6 Nov 2018 12:04:25 +0000 (13:04 +0100)]
Fixed 404 response on non-existant benchmark URL

5 months agoUpdated dependencies
Sybren A. Stüvel [Tue, 6 Nov 2018 11:58:48 +0000 (12:58 +0100)]
Updated dependencies

6 months agoAdditional explanation of raw data
Francesco Siddi [Sun, 21 Oct 2018 14:41:29 +0000 (16:41 +0200)]
Additional explanation of raw data

6 months agoTweak chart description in benchmark view
Francesco Siddi [Sun, 21 Oct 2018 12:10:53 +0000 (14:10 +0200)]
Tweak chart description in benchmark view

6 months agoSimplify building list from dict keys
Francesco Siddi [Sun, 9 Sep 2018 18:50:51 +0000 (20:50 +0200)]
Simplify building list from dict keys

6 months agoDocs and signature for query_average_render_times
Francesco Siddi [Sun, 9 Sep 2018 18:31:51 +0000 (20:31 +0200)]
Docs and signature for query_average_render_times

6 months agoLimit groups according to the current sample
Francesco Siddi [Sun, 9 Sep 2018 17:31:47 +0000 (19:31 +0200)]
Limit groups according to the current sample

6 months agoRespect line length
Francesco Siddi [Sun, 9 Sep 2018 17:31:18 +0000 (19:31 +0200)]
Respect line length

6 months agoFix typo
Francesco Siddi [Sun, 9 Sep 2018 17:31:00 +0000 (19:31 +0200)]
Fix typo

6 months agoFix name display to conform to new sample schema
Francesco Siddi [Sun, 9 Sep 2018 16:59:40 +0000 (18:59 +0200)]
Fix name display to conform to new sample schema

6 months agoAdd chart description
Francesco Siddi [Fri, 24 Aug 2018 13:24:51 +0000 (15:24 +0200)]
Add chart description

6 months agoIntroducing benchmark result comparison
Francesco Siddi [Fri, 24 Aug 2018 10:43:12 +0000 (12:43 +0200)]
Introducing benchmark result comparison

When viewing a benchmark, the results are compared
with the average results on the platform. This happens
in a chart which groups them by scene name.

6 months agoWIP on chart horizontal bar grouped
Francesco Siddi [Thu, 23 Aug 2018 15:57:40 +0000 (17:57 +0200)]
WIP on chart horizontal bar grouped

7 months agoPredefined Searches fixture: updated memory-per-scene description
Sybren A. Stüvel [Thu, 30 Aug 2018 08:25:19 +0000 (10:25 +0200)]
Predefined Searches fixture: updated memory-per-scene description

The description in the fixture is now what's currently used in production.

7 months agoUpdated predefined queries for schema v2
Sybren A. Stüvel [Thu, 30 Aug 2018 08:24:41 +0000 (10:24 +0200)]
Updated predefined queries for schema v2

This means matching on `data.device_info.compute_devices.name.keyword`
instead of `data.device_info.compute_devices.keyword`

7 months agoHandle benchmark schema version 0 as alias for version 1
Sybren A. Stüvel [Wed, 29 Aug 2018 06:58:23 +0000 (08:58 +0200)]
Handle benchmark schema version 0 as alias for version 1

Benchmark Client 1.0beta1 sent schema_version=0 even though the payload was
compatible with version 1.

7 months agoDon't repeat type in JSON `device_info.compute_devices`
Sybren A. Stüvel [Tue, 28 Aug 2018 12:55:42 +0000 (14:55 +0200)]
Don't repeat type in JSON `device_info.compute_devices`

7 months agoOverride `loaddata` management command to disable Elastic indexing
Sybren A. Stüvel [Tue, 28 Aug 2018 12:28:48 +0000 (14:28 +0200)]
Override `loaddata` management command to disable Elastic indexing

ElasticSearch indexing is very slow when using `loaddata`, as every
document is individually `PUT`. Contrast this to the `reindex` command,
which uses the bulk API to index ±1000 documents per HTTP request.

With this override, you need to manually reindex the benchmarks after
loading them.

7 months agoReindex: disable logging each POST request to ElasticSearch
Sybren A. Stüvel [Tue, 28 Aug 2018 12:10:10 +0000 (14:10 +0200)]
Reindex: disable logging each POST request to ElasticSearch

This can be re-enabled by passing `-v3` or `--verbosity 3` on the CLI.

7 months agoSpeed up schema upgrades by not pushing upgraded docs to ElasticSearch
Sybren A. Stüvel [Tue, 28 Aug 2018 12:09:42 +0000 (14:09 +0200)]
Speed up schema upgrades by not pushing upgraded docs to ElasticSearch

This means that a reindex of ElasticSearch is required after schema
upgrades.

Also report on progress of schema upgrades.

7 months agoAdded Raven to Pipfile for Sentry.io integration
Sybren A. Stüvel [Tue, 28 Aug 2018 11:08:37 +0000 (13:08 +0200)]
Added Raven to Pipfile for Sentry.io integration

7 months agoUse pathlib and BASE_DIR in example Raven configuration
Sybren A. Stüvel [Tue, 28 Aug 2018 10:59:52 +0000 (12:59 +0200)]
Use pathlib and BASE_DIR in example Raven configuration

7 months agoexample.py: removed deletion query + other refactors
Sybren A. Stüvel [Tue, 28 Aug 2018 09:11:52 +0000 (11:11 +0200)]
example.py: removed deletion query + other refactors

- pathlib instead of os.path
- Bearer instead of Token in Authorization header
- `urljoin` instead of string concatenation
- GET the URL from Location header instead of constructing manually
- Load `benchmark-v2.json`
- Use `pprint` to pretty-print the response

7 months agoUpdated README for new cmd to reindex + corrected fixture loading
Sybren A. Stüvel [Tue, 28 Aug 2018 08:05:19 +0000 (10:05 +0200)]
Updated README for new cmd to reindex + corrected fixture loading

7 months agoIntroduce Benchmark schema v2
Sybren A. Stüvel [Fri, 24 Aug 2018 15:21:39 +0000 (17:21 +0200)]
Introduce Benchmark schema v2

This commit also introduces benchmark schema version 2, plus the tools
needed to upgrade schemas.

Note that the schema version is stored outside the 'data' key; the
client posts:

{'schema_version': N, 'data': {benchmark data}}

- The schema is upgraded when a new benchmark is POSTed.
- Existing benchmarks can be upgraded using the `schema_upgrade`
  management command. Benchmarks are automatically reindexed when
  upgrading. Note that this may cause indexing errors, for example when
  ElasticSearch configured its index for schema v1, pushing a v2
  benchmark will fail.
- Reindexing does not upgrade schemas, but takes the data as-is from
  PostgreSQL.

The best approach to upgrade the schema is probably to drop the
ElasticSearch index altogether, then upgrade & reindex everything:

    pipenv run ./manage.py reindex --only-drop
    pipenv run ./manage.py schema_upgrade
    pipenv run ./manage.py reindex --no-drop

In the future we may want to make this more efficient by combining those
three steps in one, and preventing the double indexing of upgraded
benchmarks.

Note that there is no race condition between `reindex --only-drop`,
`schema_upgrade`, and users pushing new benchmarks; the pushed benchmark
will be upgraded automatically to the new schema, so ElasticSearch will
be expecting the new schema when `schema_upgrade` runs.

7 months agoKeep a record of the schema changes
Sybren A. Stüvel [Fri, 24 Aug 2018 13:10:43 +0000 (15:10 +0200)]
Keep a record of the schema changes

7 months agoExample.org URL in schema ID doesn't make sense
Sybren A. Stüvel [Fri, 24 Aug 2018 13:10:04 +0000 (15:10 +0200)]
Example.org URL in schema ID doesn't make sense

7 months agoReindexing no longer creates duplicates
Sybren A. Stüvel [Tue, 28 Aug 2018 09:22:20 +0000 (11:22 +0200)]
Reindexing no longer creates duplicates

See 2bca61883928f9c7ebe90c733766bcd127050d9c.

There is still the possibility of old data remaining in the index (when
a benchmark was deleted from the PostgreSQL database but not from Elastic
for some reason, it'll still be around after reindexing with `--no-drop`)

7 months agoreindex command: changed `--only` → `--only-drop`
Sybren A. Stüvel [Tue, 28 Aug 2018 09:21:16 +0000 (11:21 +0200)]
reindex command: changed `--only` → `--only-drop`

This makes it symmetrical with `--no-drop` and makes it explicit what's
the 'only' thing that's done.

7 months agoRestore predefined IDs for ElasticSearch documents
Sybren A. Stüvel [Tue, 28 Aug 2018 08:19:06 +0000 (10:19 +0200)]
Restore predefined IDs for ElasticSearch documents

This ID is stable, which means that when the benchmark is reindexed, it
will overwrite previously existing documents with the same ID.

This solves the 'new documents are created upon reindexing' problem.

7 months agoREADME.md: Open Data doesn't require access to the `mydata` database.
Sybren A. Stüvel [Tue, 28 Aug 2018 08:01:00 +0000 (10:01 +0200)]
README.md: Open Data doesn't require access to the `mydata` database.

8 months agoFix benchmarks fixture
Francesco Siddi [Sat, 25 Aug 2018 18:16:01 +0000 (20:16 +0200)]
Fix benchmarks fixture

Use the correct “result” type, which is a string.

8 months agoTweaks to README.md
Francesco Siddi [Sat, 25 Aug 2018 17:41:32 +0000 (19:41 +0200)]
Tweaks to README.md

8 months agoCreated test that shows problem with editing benchmarks
Sybren A. Stüvel [Fri, 24 Aug 2018 15:31:38 +0000 (17:31 +0200)]
Created test that shows problem with editing benchmarks

Saving a benchmark that's already in the psql database will create new
documents in ElasticSearch, without cleaning up the old ones.

8 months agoRedirects app now returns a 307 Temporary Redirect
Sybren A. Stüvel [Thu, 23 Aug 2018 11:41:11 +0000 (13:41 +0200)]
Redirects app now returns a 307 Temporary Redirect

Previously it sent a 301 Permanent Redirect, which breaks our ever-changing
'/download-latest' redirect.

8 months agoElastic Dump is now a ZIP with {README,LICENSE}.txt files in there too
Sybren A. Stüvel [Thu, 23 Aug 2018 11:07:29 +0000 (13:07 +0200)]
Elastic Dump is now a ZIP with {README,LICENSE}.txt files in there too

8 months agoRenamed setting DOWNLOADS_URL → ELASTIC_DUMP_URL
Sybren A. Stüvel [Thu, 23 Aug 2018 10:25:24 +0000 (12:25 +0200)]
Renamed setting DOWNLOADS_URL → ELASTIC_DUMP_URL

Even though the Elastic dumps are the only downloads we offer at the
moment, it more tightly links this setting to ELASTIC_DUMP_PATH.

8 months agoFormatting
Sybren A. Stüvel [Thu, 23 Aug 2018 10:06:28 +0000 (12:06 +0200)]
Formatting

8 months agoGracefully handle non-existing ElasticSearch indices
Sybren A. Stüvel [Thu, 23 Aug 2018 10:06:24 +0000 (12:06 +0200)]
Gracefully handle non-existing ElasticSearch indices

8 months agoIntegrate `elastic_dump` better with the entire site
Sybren A. Stüvel [Thu, 23 Aug 2018 09:51:29 +0000 (11:51 +0200)]
Integrate `elastic_dump` better with the entire site

- The command now uses Django settings to determine where to write to.
- It bzip2's the file itself, rather than writing to stdout.
- A redirect to the latest download is created/updated.

8 months agoFormatting & removal of unused import
Sybren A. Stüvel [Thu, 23 Aug 2018 09:27:06 +0000 (11:27 +0200)]
Formatting & removal of unused import

8 months agoMoved some code around to make it easier to use from other places
Sybren A. Stüvel [Thu, 23 Aug 2018 09:26:52 +0000 (11:26 +0200)]
Moved some code around to make it easier to use from other places

8 months agoAdded management command to dump ElasticSearch contents
Sybren A. Stüvel [Wed, 22 Aug 2018 14:51:49 +0000 (16:51 +0200)]
Added management command to dump ElasticSearch contents

The command dumps to stdout, one line of JSON per indexed document.

8 months agoRemoved the Django Rest Framework
Sybren A. Stüvel [Wed, 22 Aug 2018 13:57:59 +0000 (15:57 +0200)]
Removed the Django Rest Framework

- Rather than having a database table of acceptable tokens with only one
  entry in it, we now configure the token in the
  BLENDER_MYDATA['AUTH_TOKEN'] setting. Be sure to update your settings
  to include that, or benchmark submission will fail.
- I've removed some unused views and added some more unit tests.
- The schema JSON is now loaded only on demand and cached.

8 months agoAdded `odb` namespace for `opendata_benchmark` views
Sybren A. Stüvel [Wed, 22 Aug 2018 10:14:04 +0000 (12:14 +0200)]
Added `odb` namespace for `opendata_benchmark` views

This makes it more explicit which views are from which module.

8 months agoUpdate predefined search fixture for fastest CPU
Francesco Siddi [Wed, 22 Aug 2018 11:05:19 +0000 (13:05 +0200)]
Update predefined search fixture for fastest CPU

The query now filters by device_type CPU

8 months agoPredefined search fixture for fastest GPU
Francesco Siddi [Wed, 22 Aug 2018 11:01:24 +0000 (13:01 +0200)]
Predefined search fixture for fastest GPU

8 months agoFixed link web → opendata_benchmarks
Sybren A. Stüvel [Wed, 22 Aug 2018 09:55:33 +0000 (11:55 +0200)]
Fixed link web → opendata_benchmarks

8 months agoRemoved deployment instructions from README.md
Sybren A. Stüvel [Wed, 22 Aug 2018 09:50:39 +0000 (11:50 +0200)]
Removed deployment instructions from README.md

Deployment is part of the mydata project at the moment.

8 months agoMoved reindex mngmt command opendata_{main → benchmarks}
Sybren A. Stüvel [Wed, 22 Aug 2018 09:37:06 +0000 (11:37 +0200)]
Moved reindex mngmt command opendata_{main → benchmarks}

8 months agoRenamed 'web' module to 'opendata_benchmarks'
Sybren A. Stüvel [Tue, 21 Aug 2018 09:13:42 +0000 (11:13 +0200)]
Renamed 'web' module to 'opendata_benchmarks'

This change requires you to delete your `__pycache__` directories.

I've also added & updated migrations to apply the web →
opendata_benchmarks rename to the DB. See
https://stackoverflow.com/a/45007732/875379 for an explanation of this
approach:

- Updated the table-creating migrations to explicitly create `web_xxx`
  tables.
- Updated existing `opendata_benchmarks` migrations to replace the
  pre-existing `web` migrations. This way the migrations aren't applied
  twice.
- Added a migration to rename the tables and Django content types to
  `opendata_xxx`.

8 months agoBetter error message when XHR'ing chart data fails
Sybren A. Stüvel [Wed, 22 Aug 2018 08:48:34 +0000 (10:48 +0200)]
Better error message when XHR'ing chart data fails

This reuses the same function we use in Pillar and Blender ID to extract
a meaningful message from an error response.

8 months agoRefactored line drawing query & JS
Sybren A. Stüvel [Wed, 22 Aug 2018 08:39:39 +0000 (10:39 +0200)]
Refactored line drawing query & JS

The line query now looks like this:

```
{
  "aggs": {
    "toplevel": {   <--- top-level aggregation to split into a line per device type
      "aggs": {
        "line_points": {  <--- determines the points of each line
          "aggs": {
            "point": {
              "avg": {
                "field": "data.scene.stats.total_render_time"
              }
            }
          },
          "date_histogram": {
            "field": "data.timestamp",
            "interval": "1w",
            "time_zone": "Europe/Amsterdam",
            "min_doc_count": 1
          }
        }
      },
      "terms": {
        "size": 5,
        "field": "data.device_info.device_type.keyword",
        "order": {
          "_term": "asc"  <--- order lines by device type
        }
      }
    }
  },
  "size": 0
}

8 months agoRemove duplicated scripts_tutti task
Francesco Siddi [Wed, 22 Aug 2018 08:03:20 +0000 (10:03 +0200)]
Remove duplicated scripts_tutti task

8 months agoAdd scripts task to gulp watch
Francesco Siddi [Wed, 22 Aug 2018 08:03:09 +0000 (10:03 +0200)]
Add scripts task to gulp watch

8 months agoSupport for displaying error messages in charts
Francesco Siddi [Wed, 22 Aug 2018 07:50:05 +0000 (09:50 +0200)]
Support for displaying error messages in charts

8 months agoPredefined query: 'submissions-per-os' now actually counts submissions instead of...
Sybren A. Stüvel [Tue, 21 Aug 2018 16:02:21 +0000 (18:02 +0200)]
Predefined query: 'submissions-per-os' now actually counts submissions instead of samples

8 months agoPredefined searches: sort 'memory-per-scene' by term instead of by memory
Sybren A. Stüvel [Tue, 21 Aug 2018 15:48:52 +0000 (17:48 +0200)]
Predefined searches: sort 'memory-per-scene' by term instead of by memory

8 months agoUpdated predefined search fixtures for new indexing methods
Sybren A. Stüvel [Tue, 21 Aug 2018 14:59:19 +0000 (16:59 +0200)]
Updated predefined search fixtures for new indexing methods

8 months agoAdd 'View in API' link in Predefined Search admin
Sybren A. Stüvel [Tue, 21 Aug 2018 14:45:37 +0000 (16:45 +0200)]
Add 'View in API' link in Predefined Search admin

8 months agoRemove predefined searches from benchmarks.json fixture
Sybren A. Stüvel [Tue, 21 Aug 2018 14:45:24 +0000 (16:45 +0200)]
Remove predefined searches from benchmarks.json fixture

8 months agoCharts: use 'values' as key instead of 'devices' or '2'
Sybren A. Stüvel [Tue, 21 Aug 2018 14:31:16 +0000 (16:31 +0200)]
Charts: use 'values' as key instead of 'devices' or '2'

This requires the predefined searches to change as well, and makes them
less copy-pastable from Kibana. However, I think it's better to have
more meaningful keys.

8 months agoCLI command for clearing all Django caches
Sybren A. Stüvel [Tue, 21 Aug 2018 14:26:22 +0000 (16:26 +0200)]
CLI command for clearing all Django caches

8 months agoAdd 'Edit predefined seach' for people with that permission
Sybren A. Stüvel [Tue, 21 Aug 2018 14:20:02 +0000 (16:20 +0200)]
Add 'Edit predefined seach' for people with that permission

8 months agoDon't link to current page
Sybren A. Stüvel [Tue, 21 Aug 2018 14:19:47 +0000 (16:19 +0200)]
Don't link to current page

8 months agoRemoved superfluous `{{ block.super }}` calls
Sybren A. Stüvel [Tue, 21 Aug 2018 14:19:35 +0000 (16:19 +0200)]
Removed superfluous `{{ block.super }}` calls

8 months agoErase cache after editing a Predefined Search
Sybren A. Stüvel [Tue, 21 Aug 2018 14:18:34 +0000 (16:18 +0200)]
Erase cache after editing a Predefined Search

8 months agoAdd support for line charts
Francesco Siddi [Tue, 21 Aug 2018 13:46:24 +0000 (15:46 +0200)]
Add support for line charts

This allows the display of time series.

8 months agoFix typo and add comments
Francesco Siddi [Tue, 21 Aug 2018 13:45:00 +0000 (15:45 +0200)]
Fix typo and add comments

8 months agoClear Django caches after reindexing ElasticSearch
Sybren A. Stüvel [Tue, 21 Aug 2018 13:30:42 +0000 (15:30 +0200)]
Clear Django caches after reindexing ElasticSearch

8 months agoMade `manage.py reindex` always drop, unless `--no-drop` is given
Sybren A. Stüvel [Tue, 21 Aug 2018 13:13:37 +0000 (15:13 +0200)]
Made `manage.py reindex` always drop, unless `--no-drop` is given

It's not trivial (or maybe not even possible) to delete all indexed
documents of a single benchmark from ElasticSearch using the bulk API. As
a result, we'd have to do a deletion query for each individual benchmark
to be deleted (instead of one bulk call).

Since reindexing is unlikely to happen often, it's better to just erase
the entire index and build it up from scratch, rather than deleting and
replacing every single benchmark individually.

8 months agoTest that erasing a Benchmark doesn't erase all benchmarks from Elastic
Sybren A. Stüvel [Tue, 21 Aug 2018 12:59:31 +0000 (14:59 +0200)]
Test that erasing a Benchmark doesn't erase all benchmarks from Elastic

8 months agoUse ElasticSearch's refresh option to get synchronous behaviour in tests
Sybren A. Stüvel [Tue, 21 Aug 2018 12:57:53 +0000 (14:57 +0200)]
Use ElasticSearch's refresh option to get synchronous behaviour in tests

See [1] for more info.

Unfortunately, deletion seems to be asynchronous anyway, even when we wait
for the deletion task on Elastic to complete. If this can be solved somehow,
the BenchmarkIndexTests.wait_until_index_count() function can be deleted.

[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-refresh.html

8 months agoProperly test reindexing
Sybren A. Stüvel [Tue, 21 Aug 2018 11:04:23 +0000 (13:04 +0200)]
Properly test reindexing

- Erasing documents from the index without dropping the latter is no longer
  supported; the pre-existing code turned out not to work (yay testing).
- We now also test that the index after POSTing a new benchmark sample is
  the same as after reindexing all samples.

8 months agoAlso measure test coverage in the opendata_main package
Sybren A. Stüvel [Tue, 21 Aug 2018 11:03:02 +0000 (13:03 +0200)]
Also measure test coverage in the opendata_main package

8 months agoTurn @add_token into a proper decorator
Sybren A. Stüvel [Tue, 21 Aug 2018 10:47:03 +0000 (12:47 +0200)]
Turn @add_token into a proper decorator

- Return whatever the decorated function returns (a decorator should only
  modify what it needs to modify, regardless of what is used 'now').
- Decorate with @functools.wraps to keep docstrings etc. alive.

8 months agoExplicitly allow configuration of test indices
Sybren A. Stüvel [Tue, 21 Aug 2018 10:45:49 +0000 (12:45 +0200)]
Explicitly allow configuration of test indices

This also makes it possible for tests themselves to access the settings
and get the test index, rather than hardcoding 'test_benchmark'.

8 months agoIgnore backups
Sybren A. Stüvel [Tue, 21 Aug 2018 10:44:57 +0000 (12:44 +0200)]
Ignore backups

8 months agoJust refer to app module instead of module.app.Config in INSTALLED_APPS
Sybren A. Stüvel [Tue, 21 Aug 2018 09:23:11 +0000 (11:23 +0200)]
Just refer to app module instead of module.app.Config in INSTALLED_APPS

8 months agoAdded `--only` option to `manage.py reindex --{drop|erase}`
Sybren A. Stüvel [Fri, 17 Aug 2018 18:31:41 +0000 (20:31 +0200)]
Added `--only` option to `manage.py reindex --{drop|erase}`

This option makes the reindex command not reindex at all, and only drop
or erase. This was a bit easier to implement than adding more commands
for only dropping and only erasing.

8 months agoFixed some uglyness in the Elastic indexing
Sybren A. Stüvel [Fri, 17 Aug 2018 18:30:20 +0000 (20:30 +0200)]
Fixed some uglyness in the Elastic indexing

- Fixed bulk indexing so that we don't create a 'doc' subdoc.
- The Elastic _id field is now determined by ElasticSearch, and of no
  semantic value. The Benchmark ID is stored in the `benchmark_id` field
  as before.
- By renaming `BenchmarkIndex._index_name()` to `_get_index(...)`, it
  overrides the Document._get_index() function, and which allows us to call
  `bm_index.to_dict(include_meta=True)`. Maybe other `index=...` parameters
  can be removed now too.

8 months agoUse our own fonts for text.
Pablo Vazquez [Fri, 17 Aug 2018 15:53:05 +0000 (17:53 +0200)]
Use our own fonts for text.

Next is fonts for icons

8 months agoAdded CLI command for reindexing benchmarks in ElasticSearch
Sybren A. Stüvel [Fri, 17 Aug 2018 15:32:45 +0000 (17:32 +0200)]
Added CLI command for reindexing benchmarks in ElasticSearch

Run `manage.py reindex --help` for more info.

8 months agoRefactored the ElasticSearch indexing
Sybren A. Stüvel [Fri, 17 Aug 2018 15:06:54 +0000 (17:06 +0200)]
Refactored the ElasticSearch indexing

- ElasticSearch-related code is no longer defined on the database model;
  it was moved to the BenchmarkIndex class.
- Renamed search.py → elastic.py
- Unit test settings are now handled correctly, instead of  doing the
  'test in sys.argv' type of hack (which doesn't work at all when running
  via `py.test`).
- The GET response for the single-benchmark API view now responds with the
  PostgreSQL database contents, instead of the document in ElasticSearch.
- Each benchmark is indexed into multiple documents in ElasticSearch: one
  for each scene that was run.

8 months agoUse absolute paths in templates configuration
Sybren A. Stüvel [Fri, 17 Aug 2018 13:27:17 +0000 (15:27 +0200)]
Use absolute paths in templates configuration

This makes them work from other directories too (like the tests dir)

8 months agoRenamed test_index to test_page_index because it's about the index page
Sybren A. Stüvel [Fri, 17 Aug 2018 13:27:06 +0000 (15:27 +0200)]
Renamed test_index to test_page_index because it's about the index page

I got confused between 'test_index' and 'test_indexing', so the former
is now 'test_page_index' and the latter will become 'test_elastic'

8 months agoAdded missing migration
Sybren A. Stüvel [Fri, 17 Aug 2018 15:07:01 +0000 (17:07 +0200)]
Added missing migration

8 months agoNo bare except: clauses allowed
Sybren A. Stüvel [Fri, 17 Aug 2018 12:13:54 +0000 (14:13 +0200)]
No bare except: clauses allowed

8 months agoRemoved committed files from gitignored directory
Sybren A. Stüvel [Fri, 17 Aug 2018 12:10:58 +0000 (14:10 +0200)]
Removed committed files from gitignored directory

These should have never been committed in the first place.

8 months agoBasic admin backend customization
Sybren A. Stüvel [Fri, 17 Aug 2018 12:09:29 +0000 (14:09 +0200)]
Basic admin backend customization

Display site name in admin header templates.

8 months agoMade unit test for indexing predictable
Sybren A. Stüvel [Fri, 17 Aug 2018 12:00:57 +0000 (14:00 +0200)]
Made unit test for indexing predictable

Never rely on a fixed-length time.sleep() in a test, it's always going to
be either too short in certain edge cases (when doing something else on
the CPU while the tests are running) or cause unnecessary delays.

8 months agoRemoved unused imports
Sybren A. Stüvel [Fri, 17 Aug 2018 11:57:17 +0000 (13:57 +0200)]
Removed unused imports