Contribute to Open Source. Search issue labels to find the right project for you!

Tests and Continuous Integration

radare/rune

We have never had enough stability in rune to be able to write persistent tests for each module. Now that things are falling into place, it is critical that we have almost 100% test coverage and setup continuous integration so that future PRs are monitored better.

Updated 15/10/2017 15:37

Setup benchmarks

omtcyfz/heathen

Tracking the performance of the compiler is also an important task. Noticing regressions and finding hotspots early would save a lot of effort later on. Direct comparison to other major languages is also quite important in terms of ensuring that the language is quick and useful enough to be widely used.

Thus, there are few major directions of benchmarks tasks:

  • [ ] Setup benchmarks for the compiler itself and its internals, probably figuring out a way of creating extensive and representative examples to measure the performance; Google Benchmark library, which is already plugged as the dependency should be used for these purposes
  • [ ] Building entries for The Computer Language Benchmarks Game, which is an amazing sandbox for solving different problems while comparing the performance of the programming languages and their toolchains
  • [ ] The last important point is making the results accessible and frequently comparing performance as the development goes on
Updated 12/10/2017 10:28

Ensure code health

omtcyfz/heathen

While measuring test coverage (#1) is one way of ensuring the codebase is healthy, there are several others to be integrated into the development pipeline and plugged into the CI:

  • [ ] Enabling Static Analysis tools, such as Clang-Tidy, to frequently analyze the codebase and report inconsistencies; providing accessible reports for each successful build using after_success Travis CI section would be amazing
  • [ ] Useful tool for managing #includes: include-what-you-use, which might be eventually added to the LLVM main repo
  • [ ] Running tests with each of the following LLVM Sanitizer enabled would certainly expose some of the existing issues given the testset would be representative enough
Updated 12/10/2017 10:27

Add Windows buildots

omtcyfz/heathen

It would be nice to eventually support Windows but at this point, it is much more important to focus on the language itself and ensure proper Linux and macOS builds.

AppVeyor is a frequent choice for many Open Source projects, I assume it is the most convenient option at the moment.

There are also several obstacles to adding Windows support:

  • I do not have any developer experience on Windows and I do not have an access to a Windows machine
  • The existing development toolchain on Windows is a bit off at the moment: MSVC is crazy sometimes, the support for modern C++ is not as good, Windows lacks standard Unix utilities. However, LLVM has Windows buildbots (which are usually the ones to break after my commits to LLVM projects) and while it is the biggest dependency of the project, it is certainly nice. Clang is also available on Windows, from what I know it can be even used in VS; if it’s easy to setup Clang in AppVeyor it is certainly a bit easier because I feel like supporting MSVC would be painful, therefore only enabling Clang builds on Windows is preferable.
  • benchmark library has to be tweaked a bit on Windows, which is not really convenient

CMake, Python and Google Test seem to work on Windows without any issues known to me.

Updated 12/10/2017 10:26

Generate more documentation

omtcyfz/heathen

User documentation

Enabling user documentation is crucial for a positive experience for those willing to learn more about the project.

MkDocs can be used for generating documentation containing design notes, tutorials and other information relevant to the users from Markdown text files.

Developer documentation

It is currently possible to get Doxygen-generated docs for developer reference (Doxygen config is stored in docs/Codebase). However, it can be configured to generate more fancy stuff like dependency diagrams and get a better look.

Roadmap

  • [ ] Start adding design notes and information about the project to organized docs/Reference directory
  • [ ] Configure CMake and create MkDocs config for documentation generation
  • [ ] Add optional MkDocs dependency, update README
  • [ ] Start building both user and developer documentation in CI (probably create a single matrix entry for documentation and code coverage measurement for #1, this entry would be using Debug building type)
  • [ ] Upload documentation somewhere (e.g. using Github pages omtcyfz.github.io/heathen/docs/ if that is possible)
Updated 12/10/2017 10:25

Настрой систему резервного копирования

fidals/stroyprombeton

Список того, что нужно сохранять: - [ ] База данных - [ ] Медиаданные - [ ] Статика (?)

Желательно делать это через определенные интервалы (например, одновременно иметь в доступе дневной, недельный и месячный бекап базы данных)

Updated 13/10/2017 08:19

Investigate failing `tav` test on Jenkins

elastic/apm-agent-nodejs

Investigate failing tav tests on Jenkins.

                 -- running test "node test/instrumentation/modules/mysql/pool-release-1.js" with mysql
14:50:33 module.js:491
14:50:33     throw err;
14:50:33     ^
14:50:33 
14:50:33 Error: Cannot find module 'bignumber.js'
14:50:33     at Function.Module._resolveFilename (module.js:489:15)
14:50:33     at Function.Module._load (module.js:439:25)
14:50:33     at Function.Module._load (/app/node_modules/require-in-the-middle/index.js:22:24)
Updated 11/10/2017 14:59 1 Comments

Add project lib dependencies to bom

zeebe-io/zeebe

Problem

During the release process all sub project dependencies have to be manually adjusted. This is a cumbersome task which is error-prone and time-consuming.

Idea

  • All sub project versions are managed in the zb-bom as properties and in the dependency management section
  • All projects will have the same version at least until we hit 1.0.0 to ease the release process, so the next version for all projects will be 0.4.0, except for zb-parent and zb-build-tools
  • As no version has to be adjusted anymore we can automate the full release #474
  • The limitation is that we have to disable the snapshot check on the release process, so we have to ensure that we do not add SNAPSHOT dependencies of third party libraries

Tasks

  • [x] set all project versions to 0.4.0-SNAPSHOT (except for zb-parent)
  • [x] create zb-bom which contains the properties/dependency management to parent and use ${project.version} for sub project version
  • [x] in every project import the zb-bom with version ${project.version}
  • [x] remove properties/dependency management from sub projects
  • [x] adjust all release jobs to -DignoreSnapshots to prevent release to fail on snapshot check
  • [x] teach developers that we should never add SNAPSHOT dependencies for third party libraries
Updated 12/10/2017 12:23 3 Comments

Create a release pipeline in jenkins

zeebe-io/zeebe

The Zeebe projects have to be released in a specific order which is currently manually done. If we manage to remove the manual interaction required of changing the interproject dependency versions we can automate this process using a jenkins release pipeline.

Updated 16/10/2017 08:22 1 Comments

Start measuring test coverage

omtcyfz/heathen

Covering most part of the written code with tests is crucial for ensuring safety and stability.

There are several useful tools for continuous coverage measurements. llvm-cov can be used to emit coverage information. codecov simplifies the coverage measurement process by a significant margin.

  • [ ] Configure CMake scripts to provide an option to emit coverage information
  • [ ] Setup codecov
  • [ ] Add a badge to README
Updated 08/10/2017 15:51

Xcode 9 + Swift 4 Migration

TCA-Team/iOS

We will need to think about migrating to Xcode 9 and Swift 4 pretty soon.

Big questions: - Which deps will we have to update ourselves? - Which should run in Swift 4 and which in the 3.2 compatibility mode? - How can we guide the pod script that creates the targets to configure the correct build settings for these projects and not bump them all blindly to Swift 4?

Updated 10/10/2017 09:32 2 Comments

Mission "clean test logs" pt.2: Keep it that way

obspy/obspy

As pointed out in #1888, we need to make sure the test runner output stays clean in CI. We should catch stdout and stderr of the runtests call in Travis (and optionally Appveyor as well – Windows likely has quite some different behavior in compiled stuff) and slap a regex check for any unwanted output on it and fail CI if there’s any unwanted output. That way we can prevent new PRs cluttering our testrunner output again.

Updated 08/10/2017 20:20

Move CI to our server

fidals/shopelectro

Трэвис на беспланом серве билдит не очень. Сейчас у нас один travis ci идёт ~20 мин. Учитывая, что мы будем оперировать задачами с оценкой по 15-30 мин, это слишком долго. Кроме того, Трэвис часто отваливается по ресурсам, что плодит нам в ci рандомные ошибки. А это очень дорого, конечно же.

Исследуй разные коробки. Например travis, jenkins, team-city и тд. Нам нужно что-то простенькое, позже развернём коробку на своём серве.

Результат задачи - отчёт по исследованию здесь в комментах, решение по инструменту принято, создана новая задача по внедрению инструмента

Концы от нашего хоста бери у @duker33

Updated 16/10/2017 11:37 3 Comments

Continuous delivery

fidals/shopelectro

Создавай CD для прода. Каждый коммит в мастер должен автозаезжать в систему. Если что-то не так, сыпятся уведомлялки в слак/телеграм

Обязательно - бэкапы, сейчас они не делаются

Updated 11/10/2017 11:00 1 Comments

Autobuild local env

fidals/shopelectro

Сейчас чтобы развернуть локальную среду, нужно покопаться и страдать. Сам только вчера страдал пару бакетов (5ч)

Сделай какой-нить sh с автосозданием рабочей среды на локали. Много не хватает, вот что бросилось мне в глаза: - local.py+.env копируй из local.py.dist+.env.dist. В кач-ве ключей юзай или случайные или дефолтные значения - скачивай и разворачивай на локали боевую базу. В нашем случае так будет проще всего. Для этого можно развернуть что-то типа репы с последним дампом базы. Доступ к репе - по открытым ключам, которые нам скидывают Прогеры. Например тот же ключ, что и для Гита, отлично подойдёт. Или просто по паролю - разворачивай (или скачивай) на локали боевую статику. Ту, что во front/build - по включенной опции скачивай боевую media. Для этого подойдёт диалог/опции при запуске sh - всё это документи в нашем rst

Updated 17/10/2017 07:07

Coala's npm deps error

fidals/shopelectro

В этом билде можно найти стектрейс от коалы: https://travis-ci.org/fidals/shopelectro/jobs/282795733 У неё не догружаются или некорректно выполняются npm-модули. Всё как обычно с этим вашим npm

Вот фрагмент стектрейса Referenced from: airbnb-base Referenced from: /usr/app/src/.eslintrc Error: /node_modules/eslint-config-airbnb-base/rules/imports.js: Configuration for rule "import/no-unresolved" is invalid: Value "data["0"].caseSensitive" has additional properties.

Разрули ошибку или документируй

Updated 04/10/2017 11:40

Release from the CI/CD

flyve-mdm/flyve-mdm-android-inventory

Hi, @rafaelje These are features missed on the CI in the master branch push.

  • [ ] Generate ChangeLog
  • [ ] Update the package.json
  • [x] Release description
  • [ ] Merge back the develop branch

Please add these features asap. Thanks.

Updated 05/10/2017 01:40 1 Comments

Automatically detect constness issues in allocator types

eclipse/omr

1712 (derived from #1707 ) points out that the allocator types we’ve used before are subtly not const-correct, but accepted by most of our build compilers.

Nevertheless, we should detect when we get this wrong.

This might just be easily solved by having advanced toolchain builds where we build with Clang 5+ and GCC 7+ to detect early changes.

Updated 29/09/2017 20:48

Resetup test cluster with OS which is maintainable by team and devops

zeebe-io/zeebe
  • currently the Zeebe test cluster is using Gentoo as OS, which is only maintainable by Tim
  • with the further development we may need a test cluster which we have to adjust to specific needs (software, network config, etc.)
  • I think it would be better if the team/devops at the Berlin office feel comfortable to adjust the cluster to our needs
  • I would propose to resetup the cluster using a common OS like CentOS, Debian or Ubuntu
Updated 12/10/2017 12:23 7 Comments

Migrate CI to circleci 2.0 and workflow

DataDog/dd-trace-py

Tests have been broken into different jobs, grouped by library so that one flaky test doesn’t require us restarting the build from scratch.

The .tox directory is now cached between executions and will only be regenerated when tox.ini changes. This makes a big difference in execution time.

I’ve removed the generic contrib test environment since as far as I could tell everything is covered by the other test environments.

Since the tests are split among many jobs, I added a “sync” job that waits until all of them are complete and checks that all the test environments of the tox.ini have been executed.

Next steps to improve execution speed are: - refactor the tox file to avoid running several times the same tests (ex: redis) - cache docker images to improve tests startup time (~30 seconds/job)

Updated 14/10/2017 11:35

Automatic cache version

flyve-mdm/flyve-mdm-press-kit

Hi, @rafaelje and @hectorerb

it is necessary to change the version of the project site cache automatically when a gh-pages push is made from develop. For this it would be good to use the number of the commit since this will always be unique.

the file where the cache should be changed is a file called sw.js in a variable named CACHE_NAME.

currently looks like this:

var CACHE_NAME = '{{name}}-version-N'

Thanks.

Updated 22/09/2017 15:34 3 Comments

Branch and PR builds on ci.eclipse.org/omr

eclipse/omr

The Eclipse OMR project recently obtained some Linux PPC LE and Linux 390 machines to perform automated testing for PR and pushes to branches. The Eclipse foundation enabled a Jenkins CI server for us which we will be using to perform testing on the 2 platforms mentioned above. To help launch builds and set GitHub status for a commit we will be using a bot account named genie-omr.

When a PR is opened the new CI testing will not automatically kick off builds. To start the PR builds a committer has to put a comment in the PR with the following text genie-omr build [all,zlinux,plinux]. When this comment is added one of each of the PR builds will be started. Currently committers and whitelisted contributors can launch all of the PR builds with this command as there is no mechanism to allow a selection. In the future they may be able to select certain platforms based on the code that has changed. These builds will set a GitHub status similar to the Travis.vi and AppVeyor builds already being used by the Eclipse OMR project. You can check out the status of the commit in #1683 to see an example of how the Eclipse OMR CI updates the PR commit status. The Jenkins plugin being used for this fully supports the [skip ci] tag but since we are only launching builds via a request it should not matter right now.

When a PR is merged into a branch (currently there is only master) or a commit is pushed directly (this is not currently allowed in this project) builds will be kicked off for each of the branch builds on the projects Jenkins CI. These jobs will also update the GitHub status for the commit they test. You can see an example of what the status check looks like for these builds here if you click on the status for any recent merge commit (starting with SHA 0366145). Currently this build does not support and sort of [skip ci] tag but it is something we can incorporate over time.

Updated 03/10/2017 14:27 11 Comments

Add CI tests

samsung-cnct/kraken-logging-fluent-bit-daemonset

Currently, the CI run on this repository does not run comprehensive tests.

CI should:

  • spin up a lightweight cluster (https://github.com/samsung-cnct/k2cli#how-to-create-a-small-research-or-development-cluster)
  • deploy the fluent-bit daemonset on the cluster
  • run logs on the pods and verify that they contain docker logs, journal logs, kernel logs, etc. For reference on what type of logs we collect: #8 #9
Updated 27/09/2017 21:31 1 Comments

Improvements for Jenkins CI setup.

elastic/apm-agent-nodejs

This is a Meta Issue to collect ideas and improvement suggestions for further Improvements regarding Jenkins setup.

  • Slack Notifications on every build
  • Email Notifications on failure
  • Change how often the dependency tests are run or prioritize them lower somehow
Updated 19/09/2017 15:11 7 Comments

Fix Travis Py3.5 32bit build?

obspy/obspy

It’s currently broken and annoying for new PRs that will always get a red X for Travis.

See e.g. #1876 and e.g. https://travis-ci.org/obspy/obspy/builds/272860603?utm_source=github_status&utm_medium=notification

Updated 19/09/2017 09:11 3 Comments

automate process of generating easter egg code

samsung-cnct/kraken

<!– Thanks for filing an issue! Before hitting the button, please answer these questions.–>

Note: Please file issues for subcomponents under the appropriate repo

Component Repo
Krakenlib samsung-cnct/k2
Kraken-charts samsung-cnct/k2-charts
Kraken-tools samsung-cnct/k2-tools

What keywords did you search in Kraken issues before filing this one? (If you have found any duplicates, you should instead reply there.):


Is this a BUG REPORT or FEATURE REQUEST? (choose one): feature request <!– If this is a BUG REPORT, please: - Fill in as much of the template below as you can. If you leave out information, we can’t help you as well.

If this is a FEATURE REQUEST, please: - Describe in detail the feature/behavior/change you’d like to see.

In both cases, be ready for followup questions, and please respond in a timely manner. If we can’t reproduce a bug or think a feature already exists, we might close your issue. If we’re wrong, PLEASE feel free to reopen it and explain why. –> a currently open PR (#152) introduces a generated .go file to be placed into the cmd/ directory. This should be an automated part of the build process and probably shouldn’t be checked into github at all. Kraken version (use kraken version):

Environment:

Please be sure not to submit any confidential information (e.g. ssh keys, provider secrets, etc.) as issues are publicly accessible.

  • Environment variables (e.g. env | grep 'KRAKEN_):
  • Configuration files (~/.kraken/config.yaml or the file passed to kraken --config) :
  • Others:

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

Updated 06/09/2017 20:39

docker deploy

seanmars/notes

Ref: docker + kubernetes

  1. repo 放在 GitHub
  2. develop branch push 至 GitHub 後會 hook 到土砲 CI server
  3. CI server 啟動 docker-compose 執行 unit test
  4. 通過後跳版本號然後 push 到 master,hook 到土砲 CD server
  5. CD server 採 Google Cloud Container Builder 做好新版本 image 後存到 Google Cloud Registry
  6. kubenetes 套用新的 image 到 deployment 上跑 rolling-update

  7. 用 Docker Swarm 或 Nomad 的情境我就不熟了,應該大同小異。

  8. 土砲 CI/CD server 的原因是為了彈性和速度。試用過 Travis CI、GitLab 常 hook 不到,暖機時間又比自己架的久好幾倍,出事只能等不能修
Updated 06/09/2017 16:35

GitHub API Rate Limit

exercism/java

Sometimes, the journey test on travis-ci will fail to only test against the files changed because the rate limit has been exceeded.

{
  "message": "API rate limit exceeded for 52.54.31.11. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
  "documentation_url": "https://developer.github.com/v3/#rate-limiting"
}
Updated 05/09/2017 12:40

Set up continuous integration testing and coverage testing

wtbarnes/fiasco

It would be great for fiasco to have continuous integration testing set up so that tests can be run every time a pull request is made or updated. Travis CI would be a good choice for continuous integration testing, and it’s free for open source projects (yay MIT license!). Coveralls is a good choice to figure out which parts of the code are covered and which aren’t. I haven’t gone through this myself yet, as @SolarDrew was the one who set these up for PlasmaPy.

Also: this package sounds fantastic!

Updated 20/09/2017 05:45 5 Comments

Add WangscapeGui to Windows binary package

Wangscape/Wangscape

AppVeyor builds WangscapeGui successfully, but doesn’t include the resulting .exe in the binary package. Alter appveyor.yml to do the following: - [ ] Include Release\WangscapeGui.exe in Wangscape folder. - [ ] Include all required Qt DLLs (probably Qt5Core.dll, Qt5Gui.dll, Qt5Widgets.dll). - [ ] Add Qt license to licenses folder. - [ ] Update instructions for building on Windows (install Qt 5, make sure uic and moc are in %PATH%).

Updated 01/10/2017 11:31 11 Comments

Fix Dockerfile

OpenIPSL/OpenIPSL

I added a new account on Dockerhub to host the image for CI testing and configured it to automatically build the image based on the docker file hosted in the repository. This is hosted in the “addDockerFile” branch.

The generated docker image fails to run OMpython properly.

Error to be investigated further.

Updated 29/08/2017 11:33 2 Comments

Separate out Circle steps and give them names

JuliaLang/julia

Without this change, the Circle steps have the contents of the commands smashed together as the step names. This gives each step a clear name. To see the commands executed, just expand the step in the web UI and it’s the first thing shown.

The more important change here is that some of the larger steps are separated out into smaller, more modular steps. This avoids the Circle web UI being unable to fetch the log, which happens with the enormous logs generated by running everything in a single step.

[av skip] [bsd skip]

Updated 24/08/2017 17:19 6 Comments

Decide on whether to test update action during CI run

samsung-cnct/kraken

<!– Thanks for filing an issue! Before hitting the button, please answer these questions.–>

Is this a BUG REPORT or FEATURE REQUEST?: feature request, but more of a research/decision issue

<!– If this is a BUG REPORT, please: - Fill in as much of the template below as you can. If you leave out information, we can’t help you as well.

If this is a FEATURE REQUEST, please: - Describe in detail the feature/behavior/change you’d like to see.

In both cases, be ready for followup questions, and please respond in a timely manner. If we can’t reproduce a bug or think a feature already exists, we might close your issue. If we’re wrong, PLEASE feel free to reopen it and explain why. –>

K2cli has the ability to update nodes per the update action. Should we test this during CI?

Pros: It is a feature we offer; we should test it. Cons: It can take a very long time to test this action (roughly ten minutes per node being updated). As well, since update happens via a change in the config file, CI needs a way to alter its own config file.

Updated 06/09/2017 20:50 1 Comments

Add junit to CI

samsung-cnct/kraken

<!– Thanks for filing an issue! Before hitting the button, please answer these questions.–>

Is this a BUG REPORT or FEATURE REQUEST? feature request <!– If this is a BUG REPORT, please: - Fill in as much of the template below as you can. If you leave out information, we can’t help you as well.

If this is a FEATURE REQUEST, please: - Describe in detail the feature/behavior/change you’d like to see.

In both cases, be ready for followup questions, and please respond in a timely manner. If we can’t reproduce a bug or think a feature already exists, we might close your issue. If we’re wrong, PLEASE feel free to reopen it and explain why. –>

k2cli version (use k2cli version): Current version (1.08)

Environment: I’m not sure what this means.

What happened: Creating the first CI run for k2cli, we tried to add junit to the test run. This ultimately caused some of the unit tests in the project to fail, with no error message in the output.

What you expected to happen: The tests should have passed.

How to reproduce it (as minimally and precisely as possible): Add junit as per PR #104 in some of the earlier commits; watch Jenkins report an unstable build.

Updated 06/09/2017 20:51

ensure all-but-e2e correctly sets failure notification in github

samsung-cnct/kraken-lib

the all-but-e2e notification will only fire if the cluster comes all the way up correctly and comes all the way down correctly. this doesn’t always happen. we are left with a required notification this is pending and two notifications that have failed. we have sufficient information to know that the CI run failed but it would be best if the required CI notification wasn’t pending.

Updated 06/09/2017 21:08

Fork me on GitHub