![]() |
Not directly related to gpuowl but just a PSA to anyone trying the bleeding edge mainline kernels, ROCm fails to compile on 4.20-rc2 so don't bother trying an rc kernel until ROCm gets updated. It looks like some deprecated timing functions have been removed from the kernel at least and maybe some refactoring needs to be done, hopefully nothing major. If you don't spot the warning to check the error log it looks like ROCm installed but the HelloWorld test from README.md fails and gpuowl will fail as soon as it tries to call OpenCL. I don't think I installed the kernel incorrectly but if anyone is successfully using an rc kernel with ROCm please let me know.
[code]DKMS make.log for amdgpu-1.9-224 for kernel 4.20.0-042000rc2-generic (x86_64) Fri 16 Nov 13:48:45 GMT 2018 make: Entering directory '/usr/src/linux-headers-4.20.0-042000rc2-generic' Makefile:968: "Cannot use CONFIG_STACK_VALIDATION=y, please install libelf-dev, libelf-devel or elfutils-libelf-devel" CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_drm.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/main.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/symbols.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_fence.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_fence_array.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_kthread.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_io.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_module.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_device.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_mn.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_reservation.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/lib/chash.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/scheduler/gpu_scheduler.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_memory.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/scheduler/sched_fence.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_kms.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_drm_global.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_bitmap.o LD [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/lib/amdchash.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_pci.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_atombios.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_device.c: In function ‘kgd2kfd_interrupt’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_device.c:708:2: warning: ISO C90 forbids variable length array ‘patched_ihre’ [-Wvla] uint32_t patched_ihre[DIV_ROUND_UP( ^~~~~~~~ CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_tt.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_prime.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.c: In function ‘kfd_ioctl_get_clock_counters’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.c:837:2: error: implicit declaration of function ‘getrawmonotonic64’; did you mean ‘getrawmonotonic’? [-Werror=implicit-function-declaration] getrawmonotonic64(&time); ^~~~~~~~~~~~~~~~~ getrawmonotonic CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_topology.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.c:840:2: error: implicit declaration of function ‘get_monotonic_boottime64’; did you mean ‘getboottime64’? [-Werror=implicit-function-declaration] get_monotonic_boottime64(&time); ^~~~~~~~~~~~~~~~~~~~~~~~ getboottime64 CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_pasid.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_util.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_doorbell.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_module.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.c: In function ‘amdgpu_pmops_runtime_suspend’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.c:768:2: error: implicit declaration of function ‘vga_switcheroo_set_dynamic_switch’; did you mean ‘vga_switcheroo_process_delayed_switch’? [-Werror=implicit-function-declaration] vga_switcheroo_set_dynamic_switch(pdev, VGA_SWITCHEROO_OFF); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ vga_switcheroo_process_delayed_switch CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_flat_memory.o LD [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/amdkcl.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/atombios_crtc.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.o] Error 1 make[2]: *** Waiting for unfinished jobs.... CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_process.o LD [M] /var/lib/dkms/amdgpu/1.9-224/build/scheduler/amd-sched.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_object.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_queue.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c: In function ‘amdgpu_device_get_pcie_info’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3350:9: error: implicit declaration of function ‘drm_pcie_get_speed_cap_mask’; did you mean ‘pcie_get_speed_cap’? [-Werror=implicit-function-declaration] ret = drm_pcie_get_speed_cap_mask(adev->ddev, &mask); ^~~~~~~~~~~~~~~~~~~~~~~~~~~ pcie_get_speed_cap /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3356:15: error: ‘DRM_PCIE_SPEED_25’ undeclared (first use in this function); did you mean ‘PCIE_SPEED_2_5GT’? if (mask & DRM_PCIE_SPEED_25) ^~~~~~~~~~~~~~~~~ PCIE_SPEED_2_5GT /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3356:15: note: each undeclared identifier is reported only once for each function it appears in /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3358:15: error: ‘DRM_PCIE_SPEED_50’ undeclared (first use in this function); did you mean ‘DRM_PCIE_SPEED_25’? if (mask & DRM_PCIE_SPEED_50) ^~~~~~~~~~~~~~~~~ DRM_PCIE_SPEED_25 /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3360:15: error: ‘DRM_PCIE_SPEED_80’ undeclared (first use in this function); did you mean ‘DRM_PCIE_SPEED_50’? if (mask & DRM_PCIE_SPEED_80) ^~~~~~~~~~~~~~~~~ DRM_PCIE_SPEED_50 CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_lock.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager.o /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.c: In function ‘ttm_bo_vm_fault’: /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.c:314:10: error: implicit declaration of function ‘vm_insert_mixed’; did you mean ‘vmf_insert_mixed’? [-Werror=implicit-function-declaration] ret = vm_insert_mixed(&cvma, address, ^~~~~~~~~~~~~~~ vmf_insert_mixed /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.c:324:10: error: implicit declaration of function ‘vm_insert_pfn’; did you mean ‘vmf_insert_pfn’? [-Werror=implicit-function-declaration] ret = vm_insert_pfn(&cvma, address, pfn); ^~~~~~~~~~~~~ vmf_insert_pfn /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3367:9: error: implicit declaration of function ‘drm_pcie_get_max_link_width’; did you mean ‘drm_dp_max_link_rate’? [-Werror=implicit-function-declaration] ret = drm_pcie_get_max_link_width(adev->ddev, &mask); ^~~~~~~~~~~~~~~~~~~~~~~~~~~ drm_dp_max_link_rate CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_execbuf_util.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager_cik.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.o] Error 1 CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_page_alloc.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_manager.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.o] Error 1 make[2]: *** Waiting for unfinished jobs.... CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager_vi.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager_v9.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.o] Error 1 make[2]: *** Waiting for unfinished jobs.... scripts/Makefile.build:518: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu' failed make[1]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu] Error 2 make[1]: *** Waiting for unfinished jobs.... scripts/Makefile.build:518: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd' failed make[1]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd] Error 2 scripts/Makefile.build:518: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/ttm' failed make[1]: *** [/var/lib/dkms/amdgpu/1.9-224/build/ttm] Error 2 Makefile:1565: recipe for target '_module_/var/lib/dkms/amdgpu/1.9-224/build' failed make: *** [_module_/var/lib/dkms/amdgpu/1.9-224/build] Error 2 make: Leaving directory '/usr/src/linux-headers-4.20.0-042000rc2-generic'[/code] |
[QUOTE=preda;499437]Valerio: could you please prepare a speed comparison between "the fastest" (3.5) and "head" (5.0, with B1=0 (default)), on a FFT 5120K exponent (an exponent around 89M), using ROCm 1.9.1 if you can (i.e. not amdgpu-pro), and any GPU (probably RX580). Maybe you can also get GPU power information (reported by rocm-smi) in the two cases. Maybe switch between the different FFT 5120K variants on 5.0 and select the fastest.
Ken, if you have it handy, maybe I could get similar information from you (with these differences: not ROCm, but just specify the driver you use; and different GPU, that's fine; and use your fastest as baseline, not necessarily 3.5). I'm limited in my analysis because right now I have ONLY Vega64 to test on. Thus any perf testing I do of this problem will be partially "in the dark" if it does not manifest in the same way on Vega64. Thanks, Mihai[/QUOTE] [QUOTE=M344587487;500678]Not directly related to gpuowl but just a PSA to anyone trying the bleeding edge mainline kernels, ROCm fails to compile on 4.20-rc2 so don't bother trying an rc kernel until ROCm gets updated. It looks like some deprecated timing functions have been removed from the kernel at least and maybe some refactoring needs to be done, hopefully nothing major. If you don't spot the warning to check the error log it looks like ROCm installed but the HelloWorld test from README.md fails and gpuowl will fail as soon as it tries to call OpenCL. I don't think I installed the kernel incorrectly but if anyone is successfully using an rc kernel with ROCm please let me know. [code]DKMS make.log for amdgpu-1.9-224 for kernel 4.20.0-042000rc2-generic (x86_64) Fri 16 Nov 13:48:45 GMT 2018 make: Entering directory '/usr/src/linux-headers-4.20.0-042000rc2-generic' Makefile:968: "Cannot use CONFIG_STACK_VALIDATION=y, please install libelf-dev, libelf-devel or elfutils-libelf-devel" CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_drm.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/main.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/symbols.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_fence.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_fence_array.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_kthread.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_io.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_module.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_device.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_mn.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_reservation.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/lib/chash.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/scheduler/gpu_scheduler.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_memory.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/scheduler/sched_fence.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_kms.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_drm_global.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_bitmap.o LD [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/lib/amdchash.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_pci.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_atombios.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_device.c: In function ‘kgd2kfd_interrupt’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_device.c:708:2: warning: ISO C90 forbids variable length array ‘patched_ihre’ [-Wvla] uint32_t patched_ihre[DIV_ROUND_UP( ^~~~~~~~ CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_tt.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_prime.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.c: In function ‘kfd_ioctl_get_clock_counters’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.c:837:2: error: implicit declaration of function ‘getrawmonotonic64’; did you mean ‘getrawmonotonic’? [-Werror=implicit-function-declaration] getrawmonotonic64(&time); ^~~~~~~~~~~~~~~~~ getrawmonotonic CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_topology.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.c:840:2: error: implicit declaration of function ‘get_monotonic_boottime64’; did you mean ‘getboottime64’? [-Werror=implicit-function-declaration] get_monotonic_boottime64(&time); ^~~~~~~~~~~~~~~~~~~~~~~~ getboottime64 CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_pasid.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_util.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_doorbell.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_module.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.c: In function ‘amdgpu_pmops_runtime_suspend’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.c:768:2: error: implicit declaration of function ‘vga_switcheroo_set_dynamic_switch’; did you mean ‘vga_switcheroo_process_delayed_switch’? [-Werror=implicit-function-declaration] vga_switcheroo_set_dynamic_switch(pdev, VGA_SWITCHEROO_OFF); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ vga_switcheroo_process_delayed_switch CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_flat_memory.o LD [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/amdkcl.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/atombios_crtc.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.o] Error 1 make[2]: *** Waiting for unfinished jobs.... CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_process.o LD [M] /var/lib/dkms/amdgpu/1.9-224/build/scheduler/amd-sched.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_object.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_queue.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c: In function ‘amdgpu_device_get_pcie_info’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3350:9: error: implicit declaration of function ‘drm_pcie_get_speed_cap_mask’; did you mean ‘pcie_get_speed_cap’? [-Werror=implicit-function-declaration] ret = drm_pcie_get_speed_cap_mask(adev->ddev, &mask); ^~~~~~~~~~~~~~~~~~~~~~~~~~~ pcie_get_speed_cap /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3356:15: error: ‘DRM_PCIE_SPEED_25’ undeclared (first use in this function); did you mean ‘PCIE_SPEED_2_5GT’? if (mask & DRM_PCIE_SPEED_25) ^~~~~~~~~~~~~~~~~ PCIE_SPEED_2_5GT /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3356:15: note: each undeclared identifier is reported only once for each function it appears in /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3358:15: error: ‘DRM_PCIE_SPEED_50’ undeclared (first use in this function); did you mean ‘DRM_PCIE_SPEED_25’? if (mask & DRM_PCIE_SPEED_50) ^~~~~~~~~~~~~~~~~ DRM_PCIE_SPEED_25 /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3360:15: error: ‘DRM_PCIE_SPEED_80’ undeclared (first use in this function); did you mean ‘DRM_PCIE_SPEED_50’? if (mask & DRM_PCIE_SPEED_80) ^~~~~~~~~~~~~~~~~ DRM_PCIE_SPEED_50 CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_lock.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager.o /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.c: In function ‘ttm_bo_vm_fault’: /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.c:314:10: error: implicit declaration of function ‘vm_insert_mixed’; did you mean ‘vmf_insert_mixed’? [-Werror=implicit-function-declaration] ret = vm_insert_mixed(&cvma, address, ^~~~~~~~~~~~~~~ vmf_insert_mixed /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.c:324:10: error: implicit declaration of function ‘vm_insert_pfn’; did you mean ‘vmf_insert_pfn’? [-Werror=implicit-function-declaration] ret = vm_insert_pfn(&cvma, address, pfn); ^~~~~~~~~~~~~ vmf_insert_pfn /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3367:9: error: implicit declaration of function ‘drm_pcie_get_max_link_width’; did you mean ‘drm_dp_max_link_rate’? [-Werror=implicit-function-declaration] ret = drm_pcie_get_max_link_width(adev->ddev, &mask); ^~~~~~~~~~~~~~~~~~~~~~~~~~~ drm_dp_max_link_rate CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_execbuf_util.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager_cik.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.o] Error 1 CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_page_alloc.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_manager.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.o] Error 1 make[2]: *** Waiting for unfinished jobs.... CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager_vi.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager_v9.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.o] Error 1 make[2]: *** Waiting for unfinished jobs.... scripts/Makefile.build:518: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu' failed make[1]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu] Error 2 make[1]: *** Waiting for unfinished jobs.... scripts/Makefile.build:518: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd' failed make[1]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd] Error 2 scripts/Makefile.build:518: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/ttm' failed make[1]: *** [/var/lib/dkms/amdgpu/1.9-224/build/ttm] Error 2 Makefile:1565: recipe for target '_module_/var/lib/dkms/amdgpu/1.9-224/build' failed make: *** [_module_/var/lib/dkms/amdgpu/1.9-224/build] Error 2 make: Leaving directory '/usr/src/linux-headers-4.20.0-042000rc2-generic'[/code][/QUOTE] Exact. I have compiled kernel 4.20-rc1 successfully, but then it fails to load the ROCm modules. |
GpuOwl cpp sanity check fails
[QUOTE=preda;499563]Yes it makes sense. I'll look into implementing that.[/QUOTE]
[CODE]# cppcheck --enable=all --quiet . [GCD.cpp:38]: (warning) Assert statement calls a function which may have desired side effects: 'isOngoing'. [GCD.h:11]: (style) The class 'GCD' does not have a constructor. [kernel.h:31]: (warning) Member variable 'Kernel::timeSum' is not initialized in the constructor. [kernel.h:31]: (warning) Member variable 'Kernel::nCalls' is not initialized in the constructor. [clwrap.h:78]: (style) Class 'Queue' has a constructor with 1 argument that is not explicit. [Primes.h:18]: (style) Class 'Primes' has a constructor with 1 argument that is not explicit. [./Result.cpp:24]: (information) Skipping configuration 'REV' since the value of 'REV' is unknown. Use -D if you want to check it. You can use -U to skip it explicitly. [Worktodo.cpp:26]: (warning) %d in format string (no. 2) requires 'int *' but the argument type is 'unsigned int *'. [common.cpp:16]: (warning) Return value of function fopen() is not used. [common.cpp:16]: (error) Return value of allocation function 'fopen' is not stored. [./gpuowl.cpp:13]: (information) Skipping configuration 'REV' since the value of 'REV' is unknown. Use -D if you want to check it. You can use -U to skip it explicitly. (information) Cppcheck cannot find all the include files (use --check-config for details) [/CODE] |
[QUOTE=SELROC;500720][CODE]# cppcheck --enable=all --quiet .
[GCD.cpp:38]: (warning) Assert statement calls a function which may have desired side effects: 'isOngoing'. [GCD.h:11]: (style) The class 'GCD' does not have a constructor. [kernel.h:31]: (warning) Member variable 'Kernel::timeSum' is not initialized in the constructor. [kernel.h:31]: (warning) Member variable 'Kernel::nCalls' is not initialized in the constructor. [clwrap.h:78]: (style) Class 'Queue' has a constructor with 1 argument that is not explicit. [Primes.h:18]: (style) Class 'Primes' has a constructor with 1 argument that is not explicit. [./Result.cpp:24]: (information) Skipping configuration 'REV' since the value of 'REV' is unknown. Use -D if you want to check it. You can use -U to skip it explicitly. [Worktodo.cpp:26]: (warning) %d in format string (no. 2) requires 'int *' but the argument type is 'unsigned int *'. [common.cpp:16]: (warning) Return value of function fopen() is not used. [common.cpp:16]: (error) Return value of allocation function 'fopen' is not stored. [./gpuowl.cpp:13]: (information) Skipping configuration 'REV' since the value of 'REV' is unknown. Use -D if you want to check it. You can use -U to skip it explicitly. (information) Cppcheck cannot find all the include files (use --check-config for details) [/CODE][/QUOTE] Thanks, I suspect this is nothing serious, but worth fixing. Unfortunately I am currently travelling without access to a developer machine. I'll look into it when I get access. |
[QUOTE=preda;500823]Thanks, I suspect this is nothing serious, but worth fixing. Unfortunately I am currently travelling without access to a developer machine. I'll look into it when I get access.[/QUOTE]
I am currently trying to get ROCm and GpuOwl loaded into debian repositories. Running cppcheck is a necessary step for debianization to be successful. |
[QUOTE=SELROC;500973]I am currently trying to get ROCm and GpuOwl loaded into debian repositories.
Running cppcheck is a necessary step for debianization to be successful.[/QUOTE] Wow thanks! I'll look into this as soon as I can. |
[QUOTE=preda;501123]Wow thanks! I'll look into this as soon as I can.[/QUOTE]
Not without difficulty for me as a new debian packager, I need to find the exact syntax of .dsc file in order to build a debian source package. |
1 Attachment(s)
[QUOTE=preda;501123]Wow thanks! I'll look into this as soon as I can.[/QUOTE]
First attempt to build a Debian source package with [CODE]dpkg-source --build . [/CODE] |
[QUOTE=M344587487;500678]Not directly related to gpuowl but just a PSA to anyone trying the bleeding edge mainline kernels, ROCm fails to compile on 4.20-rc2 so don't bother trying an rc kernel until ROCm gets updated. It looks like some deprecated timing functions have been removed from the kernel at least and maybe some refactoring needs to be done, hopefully nothing major. If you don't spot the warning to check the error log it looks like ROCm installed but the HelloWorld test from README.md fails and gpuowl will fail as soon as it tries to call OpenCL. I don't think I installed the kernel incorrectly but if anyone is successfully using an rc kernel with ROCm please let me know.
[code]DKMS make.log for amdgpu-1.9-224 for kernel 4.20.0-042000rc2-generic (x86_64) Fri 16 Nov 13:48:45 GMT 2018 make: Entering directory '/usr/src/linux-headers-4.20.0-042000rc2-generic' Makefile:968: "Cannot use CONFIG_STACK_VALIDATION=y, please install libelf-dev, libelf-devel or elfutils-libelf-devel" CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_drm.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/main.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/symbols.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_fence.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_fence_array.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_kthread.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_io.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_module.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_device.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_mn.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_reservation.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/lib/chash.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/scheduler/gpu_scheduler.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_memory.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/scheduler/sched_fence.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_kms.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_drm_global.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_bitmap.o LD [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/lib/amdchash.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_pci.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_atombios.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_device.c: In function ‘kgd2kfd_interrupt’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_device.c:708:2: warning: ISO C90 forbids variable length array ‘patched_ihre’ [-Wvla] uint32_t patched_ihre[DIV_ROUND_UP( ^~~~~~~~ CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_tt.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/kcl_prime.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.c: In function ‘kfd_ioctl_get_clock_counters’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.c:837:2: error: implicit declaration of function ‘getrawmonotonic64’; did you mean ‘getrawmonotonic’? [-Werror=implicit-function-declaration] getrawmonotonic64(&time); ^~~~~~~~~~~~~~~~~ getrawmonotonic CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_topology.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.c:840:2: error: implicit declaration of function ‘get_monotonic_boottime64’; did you mean ‘getboottime64’? [-Werror=implicit-function-declaration] get_monotonic_boottime64(&time); ^~~~~~~~~~~~~~~~~~~~~~~~ getboottime64 CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_pasid.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_util.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_doorbell.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_module.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.c: In function ‘amdgpu_pmops_runtime_suspend’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.c:768:2: error: implicit declaration of function ‘vga_switcheroo_set_dynamic_switch’; did you mean ‘vga_switcheroo_process_delayed_switch’? [-Werror=implicit-function-declaration] vga_switcheroo_set_dynamic_switch(pdev, VGA_SWITCHEROO_OFF); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ vga_switcheroo_process_delayed_switch CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_flat_memory.o LD [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkcl/amdkcl.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/atombios_crtc.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_drv.o] Error 1 make[2]: *** Waiting for unfinished jobs.... CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_process.o LD [M] /var/lib/dkms/amdgpu/1.9-224/build/scheduler/amd-sched.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_object.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_queue.o /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c: In function ‘amdgpu_device_get_pcie_info’: /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3350:9: error: implicit declaration of function ‘drm_pcie_get_speed_cap_mask’; did you mean ‘pcie_get_speed_cap’? [-Werror=implicit-function-declaration] ret = drm_pcie_get_speed_cap_mask(adev->ddev, &mask); ^~~~~~~~~~~~~~~~~~~~~~~~~~~ pcie_get_speed_cap /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3356:15: error: ‘DRM_PCIE_SPEED_25’ undeclared (first use in this function); did you mean ‘PCIE_SPEED_2_5GT’? if (mask & DRM_PCIE_SPEED_25) ^~~~~~~~~~~~~~~~~ PCIE_SPEED_2_5GT /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3356:15: note: each undeclared identifier is reported only once for each function it appears in /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3358:15: error: ‘DRM_PCIE_SPEED_50’ undeclared (first use in this function); did you mean ‘DRM_PCIE_SPEED_25’? if (mask & DRM_PCIE_SPEED_50) ^~~~~~~~~~~~~~~~~ DRM_PCIE_SPEED_25 /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3360:15: error: ‘DRM_PCIE_SPEED_80’ undeclared (first use in this function); did you mean ‘DRM_PCIE_SPEED_50’? if (mask & DRM_PCIE_SPEED_80) ^~~~~~~~~~~~~~~~~ DRM_PCIE_SPEED_50 CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_lock.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager.o /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.c: In function ‘ttm_bo_vm_fault’: /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.c:314:10: error: implicit declaration of function ‘vm_insert_mixed’; did you mean ‘vmf_insert_mixed’? [-Werror=implicit-function-declaration] ret = vm_insert_mixed(&cvma, address, ^~~~~~~~~~~~~~~ vmf_insert_mixed /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.c:324:10: error: implicit declaration of function ‘vm_insert_pfn’; did you mean ‘vmf_insert_pfn’? [-Werror=implicit-function-declaration] ret = vm_insert_pfn(&cvma, address, pfn); ^~~~~~~~~~~~~ vmf_insert_pfn /var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.c:3367:9: error: implicit declaration of function ‘drm_pcie_get_max_link_width’; did you mean ‘drm_dp_max_link_rate’? [-Werror=implicit-function-declaration] ret = drm_pcie_get_max_link_width(adev->ddev, &mask); ^~~~~~~~~~~~~~~~~~~~~~~~~~~ drm_dp_max_link_rate CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_execbuf_util.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager_cik.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu/amdgpu_device.o] Error 1 CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_page_alloc.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_manager.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/ttm/ttm_bo_vm.o] Error 1 make[2]: *** Waiting for unfinished jobs.... CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager_vi.o CC [M] /var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_mqd_manager_v9.o cc1: some warnings being treated as errors scripts/Makefile.build:293: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.o' failed make[2]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd/kfd_chardev.o] Error 1 make[2]: *** Waiting for unfinished jobs.... scripts/Makefile.build:518: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu' failed make[1]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdgpu] Error 2 make[1]: *** Waiting for unfinished jobs.... scripts/Makefile.build:518: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd' failed make[1]: *** [/var/lib/dkms/amdgpu/1.9-224/build/amd/amdkfd] Error 2 scripts/Makefile.build:518: recipe for target '/var/lib/dkms/amdgpu/1.9-224/build/ttm' failed make[1]: *** [/var/lib/dkms/amdgpu/1.9-224/build/ttm] Error 2 Makefile:1565: recipe for target '_module_/var/lib/dkms/amdgpu/1.9-224/build' failed make: *** [_module_/var/lib/dkms/amdgpu/1.9-224/build] Error 2 make: Leaving directory '/usr/src/linux-headers-4.20.0-042000rc2-generic'[/code][/QUOTE] - Reworked STIBP Code Lands In Linux 4.20 To Fix The Performance [url]https://www.phoronix.com/scan.php?page=news_item&px=Fixed-STIBP-Lands-In-Linux-4.20[/url] - Benchmarking The Work-In-Progress Spectre/STIBP Code On The Way For Linux 4.20 [url]https://www.phoronix.com/scan.php?page=article&item=linux-420wip-stibp&num=1[/url] with this in mind, I keep an eye at ROCm updates. |
First P-1 factor found by GpuOwl
I think this is a first for GpuOwl, a genuine factor found during PRP-1.
[QUOTE] 2018-12-04 05:49:48 vega0 89625761 OK 9120000 10.18%; 2.41 ms/sq, 749 MULs; ETA 2d 10:40; 36dad77fac83843a (check 1.92s) 2018-12-04 05:50:14 vega0 89625761 9130000 10.19%; 2.41 ms/sq, 709 MULs; ETA 2d 10:32; 44880cedb4ce3506 2018-12-04 05:50:40 vega0 89625761 9140000 10.20%; 2.41 ms/sq, 736 MULs; ETA 2d 10:35; 131ff9126585b1de 2018-12-04 05:50:45 vega0 89625761 GCD 353351713683290534092214911 (56.55s) 2018-12-04 05:50:45 vega0 {"exponent":"89625761", "worktype":"PRP,P-1", "status":"F", "program":{"name":"gpuowl", "version":"5.0"}, "timestamp":"2018-12-03 18:50:45 UTC", "user":"preda", "computer":"vega0", "aid":"redacted", "fft-length":5242880, "factors":["353351713683290534092214911"], "b2":"40000000", "base":{"b1":"1000000", "bias":{"2":19}, "res64":"ed2da130bd266660"}} [/QUOTE] At 10% of PRP along 89625761 it found a factor! This factor is about 88.19bits in size, and can be found by P-1 with B1=234383 B2=17323057. |
[QUOTE=preda;501612]I think this is a first for GpuOwl, a genuine factor found during PRP-1.
At 10% of PRP along 89625761 it found a factor! This factor is about 88.19bits in size, and can be found by P-1 with B1=234383 B2=17323057.[/QUOTE] Yay, saved almost 90% of the PRP run time (and 100% of any future PRP DC). The server's parsing of the result looks incomplete to me: [URL]https://www.mersenne.org/report_exponent/?exp_lo=89625761&exp_hi=&full=1[/URL] shows "Factor:" as result for that, while an ordinary P-1 factor shows up as "Factor: 3326174307660372811303879 / (P-1, B1=730000, B2=14782500, E=12)" for [URL]https://www.mersenne.org/report_exponent/?exp_lo=89694721&full=1[/URL] |
| All times are UTC. The time now is 23:10. |
Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.