Started by timer
Running as SYSTEM
Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace
[SSH] script:
module load anaconda3_gpu
module load cudnn
cd python
rm -f tensorflow_test.txt
srun --nodes=1 --ntasks-per-node=1 --mem=16g \
--time=00:05:00 \
--partition=a100rome \
--account=bbmb-hydro \
--gpus-per-node=1 \
--gpu-bind=closest \
python3 cifar10gpu.py | tee tensorflow_test.txt
YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2`
echo YVALUE=$YVALUE > tensorflow_time.txt
[SSH] executing...
srun: job 84282 queued and waiting for resources
srun: job 84282 has been allocated resources
2024-10-09 07:43:07.815126: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-09 07:43:32.997069: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-10-09 07:44:07.564323: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-10-09 07:44:07.565199: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-10-09 07:44:07.565208: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-10-09 07:45:03.500190: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-09 07:45:05.550065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79030 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0
(50000, 32, 32, 3) (10000, 32, 32, 3)
Epoch 1/10
2024-10-09 07:45:13.691160: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
1/1563 [..............................] - ETA: 1:50:52 - loss: 2.3749 - accuracy: 0.0000e+00
39/1563 [..............................] - ETA: 2s - loss: 2.2832 - accuracy: 0.1378
78/1563 [>.............................] - ETA: 1s - loss: 2.1891 - accuracy: 0.1887
117/1563 [=>............................] - ETA: 1s - loss: 2.1438 - accuracy: 0.2102
156/1563 [=>............................] - ETA: 1s - loss: 2.0964 - accuracy: 0.2330
195/1563 [==>...........................] - ETA: 1s - loss: 2.0649 - accuracy: 0.2428
234/1563 [===>..........................] - ETA: 1s - loss: 2.0437 - accuracy: 0.2515
273/1563 [====>.........................] - ETA: 1s - loss: 2.0287 - accuracy: 0.2587
312/1563 [====>.........................] - ETA: 1s - loss: 2.0105 - accuracy: 0.2655
350/1563 [=====>........................] - ETA: 1s - loss: 1.9973 - accuracy: 0.2712
388/1563 [======>.......................] - ETA: 1s - loss: 1.9825 - accuracy: 0.2778
427/1563 [=======>......................] - ETA: 1s - loss: 1.9728 - accuracy: 0.2826
466/1563 [=======>......................] - ETA: 1s - loss: 1.9601 - accuracy: 0.2902
504/1563 [========>.....................] - ETA: 1s - loss: 1.9512 - accuracy: 0.2948
542/1563 [=========>....................] - ETA: 1s - loss: 1.9418 - accuracy: 0.2990
581/1563 [==========>...................] - ETA: 1s - loss: 1.9332 - accuracy: 0.3035
620/1563 [==========>...................] - ETA: 1s - loss: 1.9261 - accuracy: 0.3062
659/1563 [===========>..................] - ETA: 1s - loss: 1.9194 - accuracy: 0.3085
698/1563 [============>.................] - ETA: 1s - loss: 1.9117 - accuracy: 0.3121
737/1563 [=============>................] - ETA: 1s - loss: 1.9064 - accuracy: 0.3150
776/1563 [=============>................] - ETA: 1s - loss: 1.8997 - accuracy: 0.3175
815/1563 [==============>...............] - ETA: 0s - loss: 1.8918 - accuracy: 0.3202
854/1563 [===============>..............] - ETA: 0s - loss: 1.8875 - accuracy: 0.3226
893/1563 [================>.............] - ETA: 0s - loss: 1.8819 - accuracy: 0.3247
932/1563 [================>.............] - ETA: 0s - loss: 1.8770 - accuracy: 0.3265
970/1563 [=================>............] - ETA: 0s - loss: 1.8722 - accuracy: 0.3288
1009/1563 [==================>...........] - ETA: 0s - loss: 1.8684 - accuracy: 0.3298
1048/1563 [===================>..........] - ETA: 0s - loss: 1.8647 - accuracy: 0.3318
1087/1563 [===================>..........] - ETA: 0s - loss: 1.8612 - accuracy: 0.3333
1125/1563 [====================>.........] - ETA: 0s - loss: 1.8577 - accuracy: 0.3342
1164/1563 [=====================>........] - ETA: 0s - loss: 1.8534 - accuracy: 0.3362
1203/1563 [======================>.......] - ETA: 0s - loss: 1.8491 - accuracy: 0.3378
1242/1563 [======================>.......] - ETA: 0s - loss: 1.8447 - accuracy: 0.3396
1281/1563 [=======================>......] - ETA: 0s - loss: 1.8420 - accuracy: 0.3412
1320/1563 [========================>.....] - ETA: 0s - loss: 1.8389 - accuracy: 0.3431
1359/1563 [=========================>....] - ETA: 0s - loss: 1.8348 - accuracy: 0.3451
1398/1563 [=========================>....] - ETA: 0s - loss: 1.8307 - accuracy: 0.3463
1437/1563 [==========================>...] - ETA: 0s - loss: 1.8277 - accuracy: 0.3473
1476/1563 [===========================>..] - ETA: 0s - loss: 1.8246 - accuracy: 0.3482
1515/1563 [============================>.] - ETA: 0s - loss: 1.8219 - accuracy: 0.3492
1554/1563 [============================>.] - ETA: 0s - loss: 1.8180 - accuracy: 0.3505
1563/1563 [==============================] - 6s 1ms/step - loss: 1.8174 - accuracy: 0.3508
Epoch 2/10
1/1563 [..............................] - ETA: 2s - loss: 1.5546 - accuracy: 0.5000
40/1563 [..............................] - ETA: 1s - loss: 1.6325 - accuracy: 0.4258
79/1563 [>.............................] - ETA: 1s - loss: 1.6616 - accuracy: 0.4181
118/1563 [=>............................] - ETA: 1s - loss: 1.6704 - accuracy: 0.4100
157/1563 [==>...........................] - ETA: 1s - loss: 1.6704 - accuracy: 0.4076
196/1563 [==>...........................] - ETA: 1s - loss: 1.6684 - accuracy: 0.4117
235/1563 [===>..........................] - ETA: 1s - loss: 1.6681 - accuracy: 0.4108
274/1563 [====>.........................] - ETA: 1s - loss: 1.6623 - accuracy: 0.4126
313/1563 [=====>........................] - ETA: 1s - loss: 1.6609 - accuracy: 0.4155
352/1563 [=====>........................] - ETA: 1s - loss: 1.6636 - accuracy: 0.4145
391/1563 [======>.......................] - ETA: 1s - loss: 1.6605 - accuracy: 0.4163
430/1563 [=======>......................] - ETA: 1s - loss: 1.6605 - accuracy: 0.4157
469/1563 [========>.....................] - ETA: 1s - loss: 1.6560 - accuracy: 0.4174
508/1563 [========>.....................] - ETA: 1s - loss: 1.6578 - accuracy: 0.4175
547/1563 [=========>....................] - ETA: 1s - loss: 1.6585 - accuracy: 0.4177
586/1563 [==========>...................] - ETA: 1s - loss: 1.6595 - accuracy: 0.4189
624/1563 [==========>...................] - ETA: 1s - loss: 1.6572 - accuracy: 0.4199
663/1563 [===========>..................] - ETA: 1s - loss: 1.6565 - accuracy: 0.4198
701/1563 [============>.................] - ETA: 1s - loss: 1.6557 - accuracy: 0.4198
740/1563 [=============>................] - ETA: 1s - loss: 1.6513 - accuracy: 0.4212
779/1563 [=============>................] - ETA: 1s - loss: 1.6496 - accuracy: 0.4205
818/1563 [==============>...............] - ETA: 0s - loss: 1.6488 - accuracy: 0.4207
856/1563 [===============>..............] - ETA: 0s - loss: 1.6485 - accuracy: 0.4216
895/1563 [================>.............] - ETA: 0s - loss: 1.6487 - accuracy: 0.4213
933/1563 [================>.............] - ETA: 0s - loss: 1.6472 - accuracy: 0.4221
971/1563 [=================>............] - ETA: 0s - loss: 1.6436 - accuracy: 0.4229
1010/1563 [==================>...........] - ETA: 0s - loss: 1.6411 - accuracy: 0.4237
1048/1563 [===================>..........] - ETA: 0s - loss: 1.6392 - accuracy: 0.4242
1087/1563 [===================>..........] - ETA: 0s - loss: 1.6390 - accuracy: 0.4244
1126/1563 [====================>.........] - ETA: 0s - loss: 1.6373 - accuracy: 0.4249
1164/1563 [=====================>........] - ETA: 0s - loss: 1.6355 - accuracy: 0.4259
1203/1563 [======================>.......] - ETA: 0s - loss: 1.6343 - accuracy: 0.4268
1242/1563 [======================>.......] - ETA: 0s - loss: 1.6337 - accuracy: 0.4270
1280/1563 [=======================>......] - ETA: 0s - loss: 1.6329 - accuracy: 0.4270
1318/1563 [========================>.....] - ETA: 0s - loss: 1.6312 - accuracy: 0.4275
1357/1563 [=========================>....] - ETA: 0s - loss: 1.6307 - accuracy: 0.4279
1395/1563 [=========================>....] - ETA: 0s - loss: 1.6297 - accuracy: 0.4280
1433/1563 [==========================>...] - ETA: 0s - loss: 1.6283 - accuracy: 0.4282
1471/1563 [===========================>..] - ETA: 0s - loss: 1.6275 - accuracy: 0.4283
1510/1563 [===========================>..] - ETA: 0s - loss: 1.6276 - accuracy: 0.4282
1549/1563 [============================>.] - ETA: 0s - loss: 1.6269 - accuracy: 0.4284
1563/1563 [==============================] - 2s 1ms/step - loss: 1.6267 - accuracy: 0.4286
Epoch 3/10
1/1563 [..............................] - ETA: 2s - loss: 1.4458 - accuracy: 0.5312
40/1563 [..............................] - ETA: 1s - loss: 1.5579 - accuracy: 0.4594
79/1563 [>.............................] - ETA: 1s - loss: 1.5529 - accuracy: 0.4600
118/1563 [=>............................] - ETA: 1s - loss: 1.5601 - accuracy: 0.4539
157/1563 [==>...........................] - ETA: 1s - loss: 1.5743 - accuracy: 0.4490
196/1563 [==>...........................] - ETA: 1s - loss: 1.5823 - accuracy: 0.4447
235/1563 [===>..........................] - ETA: 1s - loss: 1.5771 - accuracy: 0.4468
274/1563 [====>.........................] - ETA: 1s - loss: 1.5728 - accuracy: 0.4483
313/1563 [=====>........................] - ETA: 1s - loss: 1.5662 - accuracy: 0.4494
352/1563 [=====>........................] - ETA: 1s - loss: 1.5586 - accuracy: 0.4501
391/1563 [======>.......................] - ETA: 1s - loss: 1.5612 - accuracy: 0.4488
430/1563 [=======>......................] - ETA: 1s - loss: 1.5615 - accuracy: 0.4499
469/1563 [========>.....................] - ETA: 1s - loss: 1.5577 - accuracy: 0.4512
508/1563 [========>.....................] - ETA: 1s - loss: 1.5568 - accuracy: 0.4520
547/1563 [=========>....................] - ETA: 1s - loss: 1.5587 - accuracy: 0.4508
587/1563 [==========>...................] - ETA: 1s - loss: 1.5561 - accuracy: 0.4523
626/1563 [===========>..................] - ETA: 1s - loss: 1.5545 - accuracy: 0.4542
665/1563 [===========>..................] - ETA: 1s - loss: 1.5531 - accuracy: 0.4550
704/1563 [============>.................] - ETA: 1s - loss: 1.5521 - accuracy: 0.4547
743/1563 [=============>................] - ETA: 1s - loss: 1.5536 - accuracy: 0.4541
782/1563 [==============>...............] - ETA: 1s - loss: 1.5533 - accuracy: 0.4541
821/1563 [==============>...............] - ETA: 0s - loss: 1.5529 - accuracy: 0.4538
860/1563 [===============>..............] - ETA: 0s - loss: 1.5529 - accuracy: 0.4540
899/1563 [================>.............] - ETA: 0s - loss: 1.5517 - accuracy: 0.4540
938/1563 [=================>............] - ETA: 0s - loss: 1.5516 - accuracy: 0.4545
977/1563 [=================>............] - ETA: 0s - loss: 1.5497 - accuracy: 0.4556
1016/1563 [==================>...........] - ETA: 0s - loss: 1.5496 - accuracy: 0.4553
1055/1563 [===================>..........] - ETA: 0s - loss: 1.5502 - accuracy: 0.4554
1094/1563 [===================>..........] - ETA: 0s - loss: 1.5508 - accuracy: 0.4553
1133/1563 [====================>.........] - ETA: 0s - loss: 1.5485 - accuracy: 0.4561
1172/1563 [=====================>........] - ETA: 0s - loss: 1.5489 - accuracy: 0.4553
1211/1563 [======================>.......] - ETA: 0s - loss: 1.5487 - accuracy: 0.4554
1250/1563 [======================>.......] - ETA: 0s - loss: 1.5480 - accuracy: 0.4555
1289/1563 [=======================>......] - ETA: 0s - loss: 1.5484 - accuracy: 0.4551
1328/1563 [========================>.....] - ETA: 0s - loss: 1.5476 - accuracy: 0.4556
1367/1563 [=========================>....] - ETA: 0s - loss: 1.5451 - accuracy: 0.4567
1406/1563 [=========================>....] - ETA: 0s - loss: 1.5450 - accuracy: 0.4563
1445/1563 [==========================>...] - ETA: 0s - loss: 1.5456 - accuracy: 0.4563
1484/1563 [===========================>..] - ETA: 0s - loss: 1.5451 - accuracy: 0.4563
1523/1563 [============================>.] - ETA: 0s - loss: 1.5447 - accuracy: 0.4568
1562/1563 [============================>.] - ETA: 0s - loss: 1.5442 - accuracy: 0.4571
1563/1563 [==============================] - 2s 1ms/step - loss: 1.5441 - accuracy: 0.4571
Epoch 4/10
1/1563 [..............................] - ETA: 2s - loss: 1.5278 - accuracy: 0.3750
40/1563 [..............................] - ETA: 1s - loss: 1.4474 - accuracy: 0.5000
79/1563 [>.............................] - ETA: 1s - loss: 1.4512 - accuracy: 0.4988
118/1563 [=>............................] - ETA: 1s - loss: 1.4695 - accuracy: 0.4883
157/1563 [==>...........................] - ETA: 1s - loss: 1.4746 - accuracy: 0.4831
196/1563 [==>...........................] - ETA: 1s - loss: 1.4725 - accuracy: 0.4855
235/1563 [===>..........................] - ETA: 1s - loss: 1.4803 - accuracy: 0.4836
273/1563 [====>.........................] - ETA: 1s - loss: 1.4843 - accuracy: 0.4811
311/1563 [====>.........................] - ETA: 1s - loss: 1.4885 - accuracy: 0.4797
349/1563 [=====>........................] - ETA: 1s - loss: 1.4809 - accuracy: 0.4802
388/1563 [======>.......................] - ETA: 1s - loss: 1.4817 - accuracy: 0.4803
427/1563 [=======>......................] - ETA: 1s - loss: 1.4814 - accuracy: 0.4797
466/1563 [=======>......................] - ETA: 1s - loss: 1.4819 - accuracy: 0.4794
505/1563 [========>.....................] - ETA: 1s - loss: 1.4848 - accuracy: 0.4779
544/1563 [=========>....................] - ETA: 1s - loss: 1.4872 - accuracy: 0.4774
583/1563 [==========>...................] - ETA: 1s - loss: 1.4874 - accuracy: 0.4784
621/1563 [==========>...................] - ETA: 1s - loss: 1.4863 - accuracy: 0.4794
659/1563 [===========>..................] - ETA: 1s - loss: 1.4849 - accuracy: 0.4808
698/1563 [============>.................] - ETA: 1s - loss: 1.4849 - accuracy: 0.4801
736/1563 [=============>................] - ETA: 1s - loss: 1.4867 - accuracy: 0.4790
775/1563 [=============>................] - ETA: 1s - loss: 1.4866 - accuracy: 0.4789
814/1563 [==============>...............] - ETA: 0s - loss: 1.4874 - accuracy: 0.4785
853/1563 [===============>..............] - ETA: 0s - loss: 1.4880 - accuracy: 0.4781
892/1563 [================>.............] - ETA: 0s - loss: 1.4879 - accuracy: 0.4790
931/1563 [================>.............] - ETA: 0s - loss: 1.4865 - accuracy: 0.4788
970/1563 [=================>............] - ETA: 0s - loss: 1.4868 - accuracy: 0.4791
1009/1563 [==================>...........] - ETA: 0s - loss: 1.4855 - accuracy: 0.4796
1048/1563 [===================>..........] - ETA: 0s - loss: 1.4850 - accuracy: 0.4797
1087/1563 [===================>..........] - ETA: 0s - loss: 1.4861 - accuracy: 0.4792
1126/1563 [====================>.........] - ETA: 0s - loss: 1.4855 - accuracy: 0.4789
1165/1563 [=====================>........] - ETA: 0s - loss: 1.4856 - accuracy: 0.4790
1203/1563 [======================>.......] - ETA: 0s - loss: 1.4847 - accuracy: 0.4796
1241/1563 [======================>.......] - ETA: 0s - loss: 1.4841 - accuracy: 0.4796
1279/1563 [=======================>......] - ETA: 0s - loss: 1.4838 - accuracy: 0.4795
1317/1563 [========================>.....] - ETA: 0s - loss: 1.4820 - accuracy: 0.4805
1356/1563 [=========================>....] - ETA: 0s - loss: 1.4816 - accuracy: 0.4806
1395/1563 [=========================>....] - ETA: 0s - loss: 1.4818 - accuracy: 0.4805
1434/1563 [==========================>...] - ETA: 0s - loss: 1.4837 - accuracy: 0.4800
1473/1563 [===========================>..] - ETA: 0s - loss: 1.4845 - accuracy: 0.4797
1512/1563 [============================>.] - ETA: 0s - loss: 1.4837 - accuracy: 0.4795
1550/1563 [============================>.] - ETA: 0s - loss: 1.4828 - accuracy: 0.4798
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4823 - accuracy: 0.4799
Epoch 5/10
1/1563 [..............................] - ETA: 2s - loss: 1.5043 - accuracy: 0.5000
40/1563 [..............................] - ETA: 2s - loss: 1.4624 - accuracy: 0.4797
78/1563 [>.............................] - ETA: 1s - loss: 1.4676 - accuracy: 0.4892
117/1563 [=>............................] - ETA: 1s - loss: 1.4492 - accuracy: 0.4960
156/1563 [=>............................] - ETA: 1s - loss: 1.4411 - accuracy: 0.4948
195/1563 [==>...........................] - ETA: 1s - loss: 1.4345 - accuracy: 0.4997
234/1563 [===>..........................] - ETA: 1s - loss: 1.4419 - accuracy: 0.4969
273/1563 [====>.........................] - ETA: 1s - loss: 1.4398 - accuracy: 0.4977
311/1563 [====>.........................] - ETA: 1s - loss: 1.4462 - accuracy: 0.4944
349/1563 [=====>........................] - ETA: 1s - loss: 1.4453 - accuracy: 0.4959
387/1563 [======>.......................] - ETA: 1s - loss: 1.4438 - accuracy: 0.4971
426/1563 [=======>......................] - ETA: 1s - loss: 1.4446 - accuracy: 0.4960
465/1563 [=======>......................] - ETA: 1s - loss: 1.4429 - accuracy: 0.4964
503/1563 [========>.....................] - ETA: 1s - loss: 1.4389 - accuracy: 0.4973
541/1563 [=========>....................] - ETA: 1s - loss: 1.4387 - accuracy: 0.4965
579/1563 [==========>...................] - ETA: 1s - loss: 1.4388 - accuracy: 0.4958
617/1563 [==========>...................] - ETA: 1s - loss: 1.4405 - accuracy: 0.4959
655/1563 [===========>..................] - ETA: 1s - loss: 1.4377 - accuracy: 0.4965
693/1563 [============>.................] - ETA: 1s - loss: 1.4357 - accuracy: 0.4975
732/1563 [=============>................] - ETA: 1s - loss: 1.4361 - accuracy: 0.4974
771/1563 [=============>................] - ETA: 1s - loss: 1.4348 - accuracy: 0.4981
809/1563 [==============>...............] - ETA: 0s - loss: 1.4331 - accuracy: 0.4987
847/1563 [===============>..............] - ETA: 0s - loss: 1.4326 - accuracy: 0.4982
885/1563 [===============>..............] - ETA: 0s - loss: 1.4323 - accuracy: 0.4981
923/1563 [================>.............] - ETA: 0s - loss: 1.4319 - accuracy: 0.4974
961/1563 [=================>............] - ETA: 0s - loss: 1.4316 - accuracy: 0.4975
999/1563 [==================>...........] - ETA: 0s - loss: 1.4312 - accuracy: 0.4977
1037/1563 [==================>...........] - ETA: 0s - loss: 1.4334 - accuracy: 0.4958
1075/1563 [===================>..........] - ETA: 0s - loss: 1.4322 - accuracy: 0.4963
1114/1563 [====================>.........] - ETA: 0s - loss: 1.4346 - accuracy: 0.4951
1153/1563 [=====================>........] - ETA: 0s - loss: 1.4376 - accuracy: 0.4936
1192/1563 [=====================>........] - ETA: 0s - loss: 1.4368 - accuracy: 0.4937
1230/1563 [======================>.......] - ETA: 0s - loss: 1.4369 - accuracy: 0.4939
1268/1563 [=======================>......] - ETA: 0s - loss: 1.4377 - accuracy: 0.4938
1306/1563 [========================>.....] - ETA: 0s - loss: 1.4378 - accuracy: 0.4938
1344/1563 [========================>.....] - ETA: 0s - loss: 1.4380 - accuracy: 0.4938
1382/1563 [=========================>....] - ETA: 0s - loss: 1.4364 - accuracy: 0.4943
1420/1563 [==========================>...] - ETA: 0s - loss: 1.4365 - accuracy: 0.4942
1459/1563 [===========================>..] - ETA: 0s - loss: 1.4364 - accuracy: 0.4944
1498/1563 [===========================>..] - ETA: 0s - loss: 1.4351 - accuracy: 0.4948
1536/1563 [============================>.] - ETA: 0s - loss: 1.4353 - accuracy: 0.4949
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4342 - accuracy: 0.4954
Epoch 6/10
1/1563 [..............................] - ETA: 2s - loss: 1.1899 - accuracy: 0.5625
39/1563 [..............................] - ETA: 2s - loss: 1.4033 - accuracy: 0.5088
77/1563 [>.............................] - ETA: 1s - loss: 1.3878 - accuracy: 0.5215
115/1563 [=>............................] - ETA: 1s - loss: 1.3889 - accuracy: 0.5190
153/1563 [=>............................] - ETA: 1s - loss: 1.3827 - accuracy: 0.5157
192/1563 [==>...........................] - ETA: 1s - loss: 1.3856 - accuracy: 0.5143
231/1563 [===>..........................] - ETA: 1s - loss: 1.3924 - accuracy: 0.5104
269/1563 [====>.........................] - ETA: 1s - loss: 1.3911 - accuracy: 0.5086
307/1563 [====>.........................] - ETA: 1s - loss: 1.3920 - accuracy: 0.5099
346/1563 [=====>........................] - ETA: 1s - loss: 1.3905 - accuracy: 0.5097
384/1563 [======>.......................] - ETA: 1s - loss: 1.3923 - accuracy: 0.5079
422/1563 [=======>......................] - ETA: 1s - loss: 1.3907 - accuracy: 0.5085
460/1563 [=======>......................] - ETA: 1s - loss: 1.3914 - accuracy: 0.5073
498/1563 [========>.....................] - ETA: 1s - loss: 1.3949 - accuracy: 0.5047
537/1563 [=========>....................] - ETA: 1s - loss: 1.3939 - accuracy: 0.5064
575/1563 [==========>...................] - ETA: 1s - loss: 1.3935 - accuracy: 0.5062
613/1563 [==========>...................] - ETA: 1s - loss: 1.3917 - accuracy: 0.5082
651/1563 [===========>..................] - ETA: 1s - loss: 1.3941 - accuracy: 0.5074
689/1563 [============>.................] - ETA: 1s - loss: 1.3936 - accuracy: 0.5077
727/1563 [============>.................] - ETA: 1s - loss: 1.3945 - accuracy: 0.5080
765/1563 [=============>................] - ETA: 1s - loss: 1.3953 - accuracy: 0.5076
804/1563 [==============>...............] - ETA: 1s - loss: 1.3954 - accuracy: 0.5071
843/1563 [===============>..............] - ETA: 0s - loss: 1.3971 - accuracy: 0.5066
881/1563 [===============>..............] - ETA: 0s - loss: 1.3969 - accuracy: 0.5062
919/1563 [================>.............] - ETA: 0s - loss: 1.3943 - accuracy: 0.5074
957/1563 [=================>............] - ETA: 0s - loss: 1.3943 - accuracy: 0.5076
995/1563 [==================>...........] - ETA: 0s - loss: 1.3956 - accuracy: 0.5073
1033/1563 [==================>...........] - ETA: 0s - loss: 1.3968 - accuracy: 0.5070
1071/1563 [===================>..........] - ETA: 0s - loss: 1.3968 - accuracy: 0.5069
1110/1563 [====================>.........] - ETA: 0s - loss: 1.3971 - accuracy: 0.5069
1149/1563 [=====================>........] - ETA: 0s - loss: 1.3966 - accuracy: 0.5076
1188/1563 [=====================>........] - ETA: 0s - loss: 1.3969 - accuracy: 0.5075
1226/1563 [======================>.......] - ETA: 0s - loss: 1.3969 - accuracy: 0.5075
1264/1563 [=======================>......] - ETA: 0s - loss: 1.3955 - accuracy: 0.5081
1303/1563 [========================>.....] - ETA: 0s - loss: 1.3943 - accuracy: 0.5087
1342/1563 [========================>.....] - ETA: 0s - loss: 1.3938 - accuracy: 0.5084
1381/1563 [=========================>....] - ETA: 0s - loss: 1.3931 - accuracy: 0.5083
1420/1563 [==========================>...] - ETA: 0s - loss: 1.3935 - accuracy: 0.5082
1459/1563 [===========================>..] - ETA: 0s - loss: 1.3932 - accuracy: 0.5086
1498/1563 [===========================>..] - ETA: 0s - loss: 1.3928 - accuracy: 0.5086
1536/1563 [============================>.] - ETA: 0s - loss: 1.3927 - accuracy: 0.5083
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3922 - accuracy: 0.5083
Epoch 7/10
1/1563 [..............................] - ETA: 1s - loss: 1.3423 - accuracy: 0.5000
39/1563 [..............................] - ETA: 2s - loss: 1.3362 - accuracy: 0.5296
77/1563 [>.............................] - ETA: 1s - loss: 1.3712 - accuracy: 0.5191
116/1563 [=>............................] - ETA: 1s - loss: 1.3565 - accuracy: 0.5242
155/1563 [=>............................] - ETA: 1s - loss: 1.3596 - accuracy: 0.5282
194/1563 [==>...........................] - ETA: 1s - loss: 1.3613 - accuracy: 0.5261
233/1563 [===>..........................] - ETA: 1s - loss: 1.3585 - accuracy: 0.5275
271/1563 [====>.........................] - ETA: 1s - loss: 1.3635 - accuracy: 0.5271
309/1563 [====>.........................] - ETA: 1s - loss: 1.3594 - accuracy: 0.5270
348/1563 [=====>........................] - ETA: 1s - loss: 1.3546 - accuracy: 0.5282
387/1563 [======>.......................] - ETA: 1s - loss: 1.3569 - accuracy: 0.5257
425/1563 [=======>......................] - ETA: 1s - loss: 1.3579 - accuracy: 0.5248
464/1563 [=======>......................] - ETA: 1s - loss: 1.3505 - accuracy: 0.5263
503/1563 [========>.....................] - ETA: 1s - loss: 1.3523 - accuracy: 0.5242
542/1563 [=========>....................] - ETA: 1s - loss: 1.3517 - accuracy: 0.5244
580/1563 [==========>...................] - ETA: 1s - loss: 1.3505 - accuracy: 0.5248
618/1563 [==========>...................] - ETA: 1s - loss: 1.3508 - accuracy: 0.5259
656/1563 [===========>..................] - ETA: 1s - loss: 1.3500 - accuracy: 0.5255
694/1563 [============>.................] - ETA: 1s - loss: 1.3508 - accuracy: 0.5256
732/1563 [=============>................] - ETA: 1s - loss: 1.3501 - accuracy: 0.5268
771/1563 [=============>................] - ETA: 1s - loss: 1.3490 - accuracy: 0.5274
810/1563 [==============>...............] - ETA: 0s - loss: 1.3497 - accuracy: 0.5274
849/1563 [===============>..............] - ETA: 0s - loss: 1.3479 - accuracy: 0.5285
887/1563 [================>.............] - ETA: 0s - loss: 1.3478 - accuracy: 0.5281
925/1563 [================>.............] - ETA: 0s - loss: 1.3489 - accuracy: 0.5269
963/1563 [=================>............] - ETA: 0s - loss: 1.3480 - accuracy: 0.5267
1001/1563 [==================>...........] - ETA: 0s - loss: 1.3472 - accuracy: 0.5266
1040/1563 [==================>...........] - ETA: 0s - loss: 1.3475 - accuracy: 0.5266
1079/1563 [===================>..........] - ETA: 0s - loss: 1.3479 - accuracy: 0.5266
1118/1563 [====================>.........] - ETA: 0s - loss: 1.3479 - accuracy: 0.5261
1157/1563 [=====================>........] - ETA: 0s - loss: 1.3480 - accuracy: 0.5264
1195/1563 [=====================>........] - ETA: 0s - loss: 1.3469 - accuracy: 0.5264
1233/1563 [======================>.......] - ETA: 0s - loss: 1.3464 - accuracy: 0.5264
1271/1563 [=======================>......] - ETA: 0s - loss: 1.3474 - accuracy: 0.5260
1309/1563 [========================>.....] - ETA: 0s - loss: 1.3490 - accuracy: 0.5257
1347/1563 [========================>.....] - ETA: 0s - loss: 1.3508 - accuracy: 0.5251
1386/1563 [=========================>....] - ETA: 0s - loss: 1.3512 - accuracy: 0.5248
1424/1563 [==========================>...] - ETA: 0s - loss: 1.3519 - accuracy: 0.5241
1462/1563 [===========================>..] - ETA: 0s - loss: 1.3540 - accuracy: 0.5232
1501/1563 [===========================>..] - ETA: 0s - loss: 1.3535 - accuracy: 0.5234
1539/1563 [============================>.] - ETA: 0s - loss: 1.3539 - accuracy: 0.5234
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3542 - accuracy: 0.5233
Epoch 8/10
1/1563 [..............................] - ETA: 2s - loss: 1.2522 - accuracy: 0.5000
40/1563 [..............................] - ETA: 2s - loss: 1.2579 - accuracy: 0.5508
78/1563 [>.............................] - ETA: 1s - loss: 1.2948 - accuracy: 0.5369
117/1563 [=>............................] - ETA: 1s - loss: 1.2915 - accuracy: 0.5390
156/1563 [=>............................] - ETA: 1s - loss: 1.3100 - accuracy: 0.5341
195/1563 [==>...........................] - ETA: 1s - loss: 1.3163 - accuracy: 0.5343
233/1563 [===>..........................] - ETA: 1s - loss: 1.3192 - accuracy: 0.5325
271/1563 [====>.........................] - ETA: 1s - loss: 1.3157 - accuracy: 0.5355
309/1563 [====>.........................] - ETA: 1s - loss: 1.3198 - accuracy: 0.5334
347/1563 [=====>........................] - ETA: 1s - loss: 1.3222 - accuracy: 0.5333
385/1563 [======>.......................] - ETA: 1s - loss: 1.3171 - accuracy: 0.5364
424/1563 [=======>......................] - ETA: 1s - loss: 1.3195 - accuracy: 0.5359
462/1563 [=======>......................] - ETA: 1s - loss: 1.3227 - accuracy: 0.5341
501/1563 [========>.....................] - ETA: 1s - loss: 1.3185 - accuracy: 0.5363
540/1563 [=========>....................] - ETA: 1s - loss: 1.3172 - accuracy: 0.5366
578/1563 [==========>...................] - ETA: 1s - loss: 1.3139 - accuracy: 0.5390
616/1563 [==========>...................] - ETA: 1s - loss: 1.3162 - accuracy: 0.5385
654/1563 [===========>..................] - ETA: 1s - loss: 1.3163 - accuracy: 0.5391
692/1563 [============>.................] - ETA: 1s - loss: 1.3147 - accuracy: 0.5401
731/1563 [=============>................] - ETA: 1s - loss: 1.3171 - accuracy: 0.5393
769/1563 [=============>................] - ETA: 1s - loss: 1.3165 - accuracy: 0.5395
808/1563 [==============>...............] - ETA: 0s - loss: 1.3153 - accuracy: 0.5404
846/1563 [===============>..............] - ETA: 0s - loss: 1.3167 - accuracy: 0.5401
884/1563 [===============>..............] - ETA: 0s - loss: 1.3174 - accuracy: 0.5397
922/1563 [================>.............] - ETA: 0s - loss: 1.3208 - accuracy: 0.5379
960/1563 [=================>............] - ETA: 0s - loss: 1.3199 - accuracy: 0.5378
998/1563 [==================>...........] - ETA: 0s - loss: 1.3196 - accuracy: 0.5372
1037/1563 [==================>...........] - ETA: 0s - loss: 1.3202 - accuracy: 0.5369
1076/1563 [===================>..........] - ETA: 0s - loss: 1.3206 - accuracy: 0.5367
1115/1563 [====================>.........] - ETA: 0s - loss: 1.3199 - accuracy: 0.5371
1154/1563 [=====================>........] - ETA: 0s - loss: 1.3198 - accuracy: 0.5373
1193/1563 [=====================>........] - ETA: 0s - loss: 1.3186 - accuracy: 0.5380
1231/1563 [======================>.......] - ETA: 0s - loss: 1.3181 - accuracy: 0.5383
1269/1563 [=======================>......] - ETA: 0s - loss: 1.3184 - accuracy: 0.5383
1307/1563 [========================>.....] - ETA: 0s - loss: 1.3173 - accuracy: 0.5389
1346/1563 [========================>.....] - ETA: 0s - loss: 1.3172 - accuracy: 0.5390
1385/1563 [=========================>....] - ETA: 0s - loss: 1.3167 - accuracy: 0.5391
1424/1563 [==========================>...] - ETA: 0s - loss: 1.3172 - accuracy: 0.5388
1462/1563 [===========================>..] - ETA: 0s - loss: 1.3179 - accuracy: 0.5386
1501/1563 [===========================>..] - ETA: 0s - loss: 1.3187 - accuracy: 0.5384
1539/1563 [============================>.] - ETA: 0s - loss: 1.3189 - accuracy: 0.5381
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3190 - accuracy: 0.5378
Epoch 9/10
1/1563 [..............................] - ETA: 1s - loss: 1.1476 - accuracy: 0.6250
39/1563 [..............................] - ETA: 2s - loss: 1.2595 - accuracy: 0.5713
77/1563 [>.............................] - ETA: 1s - loss: 1.3017 - accuracy: 0.5544
116/1563 [=>............................] - ETA: 1s - loss: 1.3063 - accuracy: 0.5501
154/1563 [=>............................] - ETA: 1s - loss: 1.3080 - accuracy: 0.5444
193/1563 [==>...........................] - ETA: 1s - loss: 1.3107 - accuracy: 0.5444
232/1563 [===>..........................] - ETA: 1s - loss: 1.3038 - accuracy: 0.5481
270/1563 [====>.........................] - ETA: 1s - loss: 1.3034 - accuracy: 0.5498
308/1563 [====>.........................] - ETA: 1s - loss: 1.3055 - accuracy: 0.5491
346/1563 [=====>........................] - ETA: 1s - loss: 1.3050 - accuracy: 0.5472
385/1563 [======>.......................] - ETA: 1s - loss: 1.3015 - accuracy: 0.5476
424/1563 [=======>......................] - ETA: 1s - loss: 1.3013 - accuracy: 0.5483
463/1563 [=======>......................] - ETA: 1s - loss: 1.3003 - accuracy: 0.5486
501/1563 [========>.....................] - ETA: 1s - loss: 1.2998 - accuracy: 0.5476
539/1563 [=========>....................] - ETA: 1s - loss: 1.2998 - accuracy: 0.5477
577/1563 [==========>...................] - ETA: 1s - loss: 1.3020 - accuracy: 0.5475
615/1563 [==========>...................] - ETA: 1s - loss: 1.3012 - accuracy: 0.5483
653/1563 [===========>..................] - ETA: 1s - loss: 1.3008 - accuracy: 0.5487
691/1563 [============>.................] - ETA: 1s - loss: 1.3006 - accuracy: 0.5483
729/1563 [============>.................] - ETA: 1s - loss: 1.2989 - accuracy: 0.5484
768/1563 [=============>................] - ETA: 1s - loss: 1.2960 - accuracy: 0.5496
806/1563 [==============>...............] - ETA: 0s - loss: 1.2936 - accuracy: 0.5499
844/1563 [===============>..............] - ETA: 0s - loss: 1.2938 - accuracy: 0.5496
882/1563 [===============>..............] - ETA: 0s - loss: 1.2923 - accuracy: 0.5502
920/1563 [================>.............] - ETA: 0s - loss: 1.2901 - accuracy: 0.5510
958/1563 [=================>............] - ETA: 0s - loss: 1.2905 - accuracy: 0.5502
997/1563 [==================>...........] - ETA: 0s - loss: 1.2898 - accuracy: 0.5510
1035/1563 [==================>...........] - ETA: 0s - loss: 1.2892 - accuracy: 0.5510
1073/1563 [===================>..........] - ETA: 0s - loss: 1.2877 - accuracy: 0.5511
1111/1563 [====================>.........] - ETA: 0s - loss: 1.2881 - accuracy: 0.5515
1149/1563 [=====================>........] - ETA: 0s - loss: 1.2881 - accuracy: 0.5513
1187/1563 [=====================>........] - ETA: 0s - loss: 1.2871 - accuracy: 0.5516
1225/1563 [======================>.......] - ETA: 0s - loss: 1.2884 - accuracy: 0.5515
1263/1563 [=======================>......] - ETA: 0s - loss: 1.2883 - accuracy: 0.5513
1301/1563 [=======================>......] - ETA: 0s - loss: 1.2887 - accuracy: 0.5510
1339/1563 [========================>.....] - ETA: 0s - loss: 1.2872 - accuracy: 0.5513
1377/1563 [=========================>....] - ETA: 0s - loss: 1.2863 - accuracy: 0.5510
1415/1563 [==========================>...] - ETA: 0s - loss: 1.2872 - accuracy: 0.5505
1453/1563 [==========================>...] - ETA: 0s - loss: 1.2872 - accuracy: 0.5505
1491/1563 [===========================>..] - ETA: 0s - loss: 1.2871 - accuracy: 0.5505
1529/1563 [============================>.] - ETA: 0s - loss: 1.2867 - accuracy: 0.5506
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2867 - accuracy: 0.5508
Epoch 10/10
1/1563 [..............................] - ETA: 2s - loss: 1.4845 - accuracy: 0.4062
39/1563 [..............................] - ETA: 2s - loss: 1.2220 - accuracy: 0.5745
78/1563 [>.............................] - ETA: 1s - loss: 1.2313 - accuracy: 0.5737
117/1563 [=>............................] - ETA: 1s - loss: 1.2350 - accuracy: 0.5686
155/1563 [=>............................] - ETA: 1s - loss: 1.2523 - accuracy: 0.5617
193/1563 [==>...........................] - ETA: 1s - loss: 1.2513 - accuracy: 0.5612
231/1563 [===>..........................] - ETA: 1s - loss: 1.2515 - accuracy: 0.5605
269/1563 [====>.........................] - ETA: 1s - loss: 1.2558 - accuracy: 0.5583
307/1563 [====>.........................] - ETA: 1s - loss: 1.2567 - accuracy: 0.5585
346/1563 [=====>........................] - ETA: 1s - loss: 1.2575 - accuracy: 0.5579
384/1563 [======>.......................] - ETA: 1s - loss: 1.2593 - accuracy: 0.5570
423/1563 [=======>......................] - ETA: 1s - loss: 1.2537 - accuracy: 0.5602
461/1563 [=======>......................] - ETA: 1s - loss: 1.2571 - accuracy: 0.5586
499/1563 [========>.....................] - ETA: 1s - loss: 1.2571 - accuracy: 0.5591
537/1563 [=========>....................] - ETA: 1s - loss: 1.2566 - accuracy: 0.5589
575/1563 [==========>...................] - ETA: 1s - loss: 1.2571 - accuracy: 0.5586
614/1563 [==========>...................] - ETA: 1s - loss: 1.2576 - accuracy: 0.5582
652/1563 [===========>..................] - ETA: 1s - loss: 1.2580 - accuracy: 0.5586
690/1563 [============>.................] - ETA: 1s - loss: 1.2600 - accuracy: 0.5585
728/1563 [============>.................] - ETA: 1s - loss: 1.2592 - accuracy: 0.5586
767/1563 [=============>................] - ETA: 1s - loss: 1.2560 - accuracy: 0.5594
806/1563 [==============>...............] - ETA: 0s - loss: 1.2564 - accuracy: 0.5584
845/1563 [===============>..............] - ETA: 0s - loss: 1.2569 - accuracy: 0.5584
884/1563 [===============>..............] - ETA: 0s - loss: 1.2567 - accuracy: 0.5581
922/1563 [================>.............] - ETA: 0s - loss: 1.2560 - accuracy: 0.5578
960/1563 [=================>............] - ETA: 0s - loss: 1.2569 - accuracy: 0.5580
998/1563 [==================>...........] - ETA: 0s - loss: 1.2577 - accuracy: 0.5580
1036/1563 [==================>...........] - ETA: 0s - loss: 1.2581 - accuracy: 0.5587
1075/1563 [===================>..........] - ETA: 0s - loss: 1.2571 - accuracy: 0.5590
1114/1563 [====================>.........] - ETA: 0s - loss: 1.2563 - accuracy: 0.5593
1153/1563 [=====================>........] - ETA: 0s - loss: 1.2567 - accuracy: 0.5592
1191/1563 [=====================>........] - ETA: 0s - loss: 1.2575 - accuracy: 0.5590
1229/1563 [======================>.......] - ETA: 0s - loss: 1.2564 - accuracy: 0.5596
1267/1563 [=======================>......] - ETA: 0s - loss: 1.2566 - accuracy: 0.5594
1305/1563 [========================>.....] - ETA: 0s - loss: 1.2556 - accuracy: 0.5598
1343/1563 [========================>.....] - ETA: 0s - loss: 1.2552 - accuracy: 0.5599
1382/1563 [=========================>....] - ETA: 0s - loss: 1.2556 - accuracy: 0.5602
1421/1563 [==========================>...] - ETA: 0s - loss: 1.2562 - accuracy: 0.5600
1460/1563 [===========================>..] - ETA: 0s - loss: 1.2548 - accuracy: 0.5609
1499/1563 [===========================>..] - ETA: 0s - loss: 1.2555 - accuracy: 0.5610
1538/1563 [============================>.] - ETA: 0s - loss: 1.2562 - accuracy: 0.5604
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2557 - accuracy: 0.5604
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten (Flatten) (None, 3072) 0
dense (Dense) (None, 3000) 9219000
dense_1 (Dense) (None, 1000) 3001000
dense_2 (Dense) (None, 10) 10010
=================================================================
Total params: 12,230,010
Trainable params: 12,230,010
Non-trainable params: 0
_________________________________________________________________
None
wall time seconds (without loading test data set)= 36.99117040634155
hydro05
[SSH] completed
[SSH] exit-status: 0
[workspace] $ /bin/sh -xe /tmp/jenkins7114624440211843418.sh
+ scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace
Recording plot data
Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt
Finished: SUCCESS
![[Jenkins]](/static/755cd4af/images/svgs/logo.svg){kind=logo}
