Started by timer
Running as SYSTEM
Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace
[SSH] script:
module load anaconda3_gpu
module load cudnn
cd python
rm -f tensorflow_test.txt
srun --nodes=1 --ntasks-per-node=1 --mem=16g \
--time=00:05:00 \
--partition=a100rome \
--account=bbmb-hydro \
--gpus-per-node=1 \
--gpu-bind=closest \
python3 cifar10gpu.py | tee tensorflow_test.txt
YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2`
echo YVALUE=$YVALUE > tensorflow_time.txt
[SSH] executing...
srun: job 90245 queued and waiting for resources
srun: job 90245 has been allocated resources
2024-10-20 07:43:06.556321: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-20 07:43:29.604896: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-10-20 07:43:57.878335: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-10-20 07:43:57.878956: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-10-20 07:43:57.878965: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-10-20 07:44:41.256901: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-10-20 07:44:43.305861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0
(50000, 32, 32, 3) (10000, 32, 32, 3)
Epoch 1/10
2024-10-20 07:44:50.228025: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
1/1563 [..............................] - ETA: 1:42:26 - loss: 2.2525 - accuracy: 0.2500
38/1563 [..............................] - ETA: 2s - loss: 2.2410 - accuracy: 0.1711
76/1563 [>.............................] - ETA: 2s - loss: 2.1797 - accuracy: 0.1945
114/1563 [=>............................] - ETA: 1s - loss: 2.1237 - accuracy: 0.2223
153/1563 [=>............................] - ETA: 1s - loss: 2.0963 - accuracy: 0.2371
191/1563 [==>...........................] - ETA: 1s - loss: 2.0723 - accuracy: 0.2472
230/1563 [===>..........................] - ETA: 1s - loss: 2.0495 - accuracy: 0.2594
269/1563 [====>.........................] - ETA: 1s - loss: 2.0283 - accuracy: 0.2717
308/1563 [====>.........................] - ETA: 1s - loss: 2.0119 - accuracy: 0.2779
347/1563 [=====>........................] - ETA: 1s - loss: 2.0002 - accuracy: 0.2830
386/1563 [======>.......................] - ETA: 1s - loss: 1.9859 - accuracy: 0.2892
424/1563 [=======>......................] - ETA: 1s - loss: 1.9739 - accuracy: 0.2929
463/1563 [=======>......................] - ETA: 1s - loss: 1.9648 - accuracy: 0.2966
502/1563 [========>.....................] - ETA: 1s - loss: 1.9555 - accuracy: 0.3000
541/1563 [=========>....................] - ETA: 1s - loss: 1.9452 - accuracy: 0.3031
580/1563 [==========>...................] - ETA: 1s - loss: 1.9354 - accuracy: 0.3066
619/1563 [==========>...................] - ETA: 1s - loss: 1.9276 - accuracy: 0.3091
658/1563 [===========>..................] - ETA: 1s - loss: 1.9218 - accuracy: 0.3124
697/1563 [============>.................] - ETA: 1s - loss: 1.9143 - accuracy: 0.3155
736/1563 [=============>................] - ETA: 1s - loss: 1.9075 - accuracy: 0.3179
774/1563 [=============>................] - ETA: 1s - loss: 1.9012 - accuracy: 0.3206
813/1563 [==============>...............] - ETA: 0s - loss: 1.8955 - accuracy: 0.3223
852/1563 [===============>..............] - ETA: 0s - loss: 1.8901 - accuracy: 0.3246
891/1563 [================>.............] - ETA: 0s - loss: 1.8853 - accuracy: 0.3266
930/1563 [================>.............] - ETA: 0s - loss: 1.8774 - accuracy: 0.3301
969/1563 [=================>............] - ETA: 0s - loss: 1.8730 - accuracy: 0.3314
1008/1563 [==================>...........] - ETA: 0s - loss: 1.8680 - accuracy: 0.3333
1047/1563 [===================>..........] - ETA: 0s - loss: 1.8644 - accuracy: 0.3346
1085/1563 [===================>..........] - ETA: 0s - loss: 1.8593 - accuracy: 0.3360
1124/1563 [====================>.........] - ETA: 0s - loss: 1.8547 - accuracy: 0.3382
1163/1563 [=====================>........] - ETA: 0s - loss: 1.8507 - accuracy: 0.3397
1202/1563 [======================>.......] - ETA: 0s - loss: 1.8452 - accuracy: 0.3413
1241/1563 [======================>.......] - ETA: 0s - loss: 1.8417 - accuracy: 0.3431
1280/1563 [=======================>......] - ETA: 0s - loss: 1.8384 - accuracy: 0.3445
1319/1563 [========================>.....] - ETA: 0s - loss: 1.8361 - accuracy: 0.3453
1358/1563 [=========================>....] - ETA: 0s - loss: 1.8336 - accuracy: 0.3465
1397/1563 [=========================>....] - ETA: 0s - loss: 1.8299 - accuracy: 0.3480
1436/1563 [==========================>...] - ETA: 0s - loss: 1.8278 - accuracy: 0.3489
1475/1563 [===========================>..] - ETA: 0s - loss: 1.8235 - accuracy: 0.3508
1514/1563 [============================>.] - ETA: 0s - loss: 1.8203 - accuracy: 0.3526
1553/1563 [============================>.] - ETA: 0s - loss: 1.8179 - accuracy: 0.3535
1563/1563 [==============================] - 6s 1ms/step - loss: 1.8178 - accuracy: 0.3535
Epoch 2/10
1/1563 [..............................] - ETA: 2s - loss: 1.9457 - accuracy: 0.1875
40/1563 [..............................] - ETA: 2s - loss: 1.6519 - accuracy: 0.4141
79/1563 [>.............................] - ETA: 1s - loss: 1.6554 - accuracy: 0.4189
118/1563 [=>............................] - ETA: 1s - loss: 1.6414 - accuracy: 0.4240
157/1563 [==>...........................] - ETA: 1s - loss: 1.6485 - accuracy: 0.4218
196/1563 [==>...........................] - ETA: 1s - loss: 1.6557 - accuracy: 0.4217
235/1563 [===>..........................] - ETA: 1s - loss: 1.6578 - accuracy: 0.4215
274/1563 [====>.........................] - ETA: 1s - loss: 1.6629 - accuracy: 0.4174
313/1563 [=====>........................] - ETA: 1s - loss: 1.6584 - accuracy: 0.4196
352/1563 [=====>........................] - ETA: 1s - loss: 1.6578 - accuracy: 0.4193
390/1563 [======>.......................] - ETA: 1s - loss: 1.6534 - accuracy: 0.4218
429/1563 [=======>......................] - ETA: 1s - loss: 1.6485 - accuracy: 0.4234
468/1563 [=======>......................] - ETA: 1s - loss: 1.6514 - accuracy: 0.4214
507/1563 [========>.....................] - ETA: 1s - loss: 1.6515 - accuracy: 0.4223
546/1563 [=========>....................] - ETA: 1s - loss: 1.6501 - accuracy: 0.4230
585/1563 [==========>...................] - ETA: 1s - loss: 1.6471 - accuracy: 0.4239
624/1563 [==========>...................] - ETA: 1s - loss: 1.6454 - accuracy: 0.4245
663/1563 [===========>..................] - ETA: 1s - loss: 1.6458 - accuracy: 0.4231
702/1563 [============>.................] - ETA: 1s - loss: 1.6454 - accuracy: 0.4232
741/1563 [=============>................] - ETA: 1s - loss: 1.6429 - accuracy: 0.4240
780/1563 [=============>................] - ETA: 1s - loss: 1.6426 - accuracy: 0.4242
819/1563 [==============>...............] - ETA: 0s - loss: 1.6433 - accuracy: 0.4233
858/1563 [===============>..............] - ETA: 0s - loss: 1.6438 - accuracy: 0.4237
897/1563 [================>.............] - ETA: 0s - loss: 1.6423 - accuracy: 0.4238
936/1563 [================>.............] - ETA: 0s - loss: 1.6397 - accuracy: 0.4244
974/1563 [=================>............] - ETA: 0s - loss: 1.6389 - accuracy: 0.4254
1012/1563 [==================>...........] - ETA: 0s - loss: 1.6388 - accuracy: 0.4251
1051/1563 [===================>..........] - ETA: 0s - loss: 1.6377 - accuracy: 0.4253
1090/1563 [===================>..........] - ETA: 0s - loss: 1.6358 - accuracy: 0.4256
1129/1563 [====================>.........] - ETA: 0s - loss: 1.6356 - accuracy: 0.4251
1168/1563 [=====================>........] - ETA: 0s - loss: 1.6340 - accuracy: 0.4262
1207/1563 [======================>.......] - ETA: 0s - loss: 1.6333 - accuracy: 0.4257
1245/1563 [======================>.......] - ETA: 0s - loss: 1.6317 - accuracy: 0.4259
1284/1563 [=======================>......] - ETA: 0s - loss: 1.6302 - accuracy: 0.4264
1323/1563 [========================>.....] - ETA: 0s - loss: 1.6287 - accuracy: 0.4269
1361/1563 [=========================>....] - ETA: 0s - loss: 1.6277 - accuracy: 0.4271
1400/1563 [=========================>....] - ETA: 0s - loss: 1.6278 - accuracy: 0.4264
1439/1563 [==========================>...] - ETA: 0s - loss: 1.6271 - accuracy: 0.4262
1477/1563 [===========================>..] - ETA: 0s - loss: 1.6272 - accuracy: 0.4262
1516/1563 [============================>.] - ETA: 0s - loss: 1.6255 - accuracy: 0.4267
1554/1563 [============================>.] - ETA: 0s - loss: 1.6247 - accuracy: 0.4269
1563/1563 [==============================] - 2s 1ms/step - loss: 1.6244 - accuracy: 0.4270
Epoch 3/10
1/1563 [..............................] - ETA: 2s - loss: 1.4461 - accuracy: 0.5938
40/1563 [..............................] - ETA: 1s - loss: 1.5781 - accuracy: 0.4430
79/1563 [>.............................] - ETA: 1s - loss: 1.5582 - accuracy: 0.4525
118/1563 [=>............................] - ETA: 1s - loss: 1.5689 - accuracy: 0.4523
157/1563 [==>...........................] - ETA: 1s - loss: 1.5669 - accuracy: 0.4498
196/1563 [==>...........................] - ETA: 1s - loss: 1.5602 - accuracy: 0.4526
234/1563 [===>..........................] - ETA: 1s - loss: 1.5638 - accuracy: 0.4503
272/1563 [====>.........................] - ETA: 1s - loss: 1.5590 - accuracy: 0.4497
310/1563 [====>.........................] - ETA: 1s - loss: 1.5616 - accuracy: 0.4490
348/1563 [=====>........................] - ETA: 1s - loss: 1.5633 - accuracy: 0.4494
387/1563 [======>.......................] - ETA: 1s - loss: 1.5610 - accuracy: 0.4503
426/1563 [=======>......................] - ETA: 1s - loss: 1.5597 - accuracy: 0.4514
465/1563 [=======>......................] - ETA: 1s - loss: 1.5581 - accuracy: 0.4508
504/1563 [========>.....................] - ETA: 1s - loss: 1.5578 - accuracy: 0.4501
543/1563 [=========>....................] - ETA: 1s - loss: 1.5558 - accuracy: 0.4511
582/1563 [==========>...................] - ETA: 1s - loss: 1.5543 - accuracy: 0.4516
621/1563 [==========>...................] - ETA: 1s - loss: 1.5558 - accuracy: 0.4511
660/1563 [===========>..................] - ETA: 1s - loss: 1.5510 - accuracy: 0.4535
699/1563 [============>.................] - ETA: 1s - loss: 1.5514 - accuracy: 0.4522
737/1563 [=============>................] - ETA: 1s - loss: 1.5508 - accuracy: 0.4521
776/1563 [=============>................] - ETA: 1s - loss: 1.5477 - accuracy: 0.4535
815/1563 [==============>...............] - ETA: 0s - loss: 1.5501 - accuracy: 0.4525
854/1563 [===============>..............] - ETA: 0s - loss: 1.5488 - accuracy: 0.4528
893/1563 [================>.............] - ETA: 0s - loss: 1.5492 - accuracy: 0.4522
932/1563 [================>.............] - ETA: 0s - loss: 1.5494 - accuracy: 0.4518
971/1563 [=================>............] - ETA: 0s - loss: 1.5507 - accuracy: 0.4518
1010/1563 [==================>...........] - ETA: 0s - loss: 1.5503 - accuracy: 0.4518
1049/1563 [===================>..........] - ETA: 0s - loss: 1.5505 - accuracy: 0.4517
1088/1563 [===================>..........] - ETA: 0s - loss: 1.5493 - accuracy: 0.4527
1127/1563 [====================>.........] - ETA: 0s - loss: 1.5477 - accuracy: 0.4535
1166/1563 [=====================>........] - ETA: 0s - loss: 1.5487 - accuracy: 0.4535
1205/1563 [======================>.......] - ETA: 0s - loss: 1.5478 - accuracy: 0.4539
1244/1563 [======================>.......] - ETA: 0s - loss: 1.5473 - accuracy: 0.4541
1282/1563 [=======================>......] - ETA: 0s - loss: 1.5464 - accuracy: 0.4543
1319/1563 [========================>.....] - ETA: 0s - loss: 1.5451 - accuracy: 0.4550
1357/1563 [=========================>....] - ETA: 0s - loss: 1.5444 - accuracy: 0.4554
1394/1563 [=========================>....] - ETA: 0s - loss: 1.5438 - accuracy: 0.4557
1432/1563 [==========================>...] - ETA: 0s - loss: 1.5432 - accuracy: 0.4560
1470/1563 [===========================>..] - ETA: 0s - loss: 1.5425 - accuracy: 0.4565
1508/1563 [===========================>..] - ETA: 0s - loss: 1.5428 - accuracy: 0.4564
1545/1563 [============================>.] - ETA: 0s - loss: 1.5432 - accuracy: 0.4566
1563/1563 [==============================] - 2s 1ms/step - loss: 1.5428 - accuracy: 0.4568
Epoch 4/10
1/1563 [..............................] - ETA: 2s - loss: 1.5236 - accuracy: 0.4375
39/1563 [..............................] - ETA: 2s - loss: 1.4631 - accuracy: 0.4784
76/1563 [>.............................] - ETA: 2s - loss: 1.4395 - accuracy: 0.4914
113/1563 [=>............................] - ETA: 1s - loss: 1.4662 - accuracy: 0.4751
151/1563 [=>............................] - ETA: 1s - loss: 1.4729 - accuracy: 0.4785
189/1563 [==>...........................] - ETA: 1s - loss: 1.4645 - accuracy: 0.4783
227/1563 [===>..........................] - ETA: 1s - loss: 1.4701 - accuracy: 0.4738
265/1563 [====>.........................] - ETA: 1s - loss: 1.4806 - accuracy: 0.4725
302/1563 [====>.........................] - ETA: 1s - loss: 1.4756 - accuracy: 0.4744
340/1563 [=====>........................] - ETA: 1s - loss: 1.4827 - accuracy: 0.4713
378/1563 [======>.......................] - ETA: 1s - loss: 1.4877 - accuracy: 0.4688
416/1563 [======>.......................] - ETA: 1s - loss: 1.4890 - accuracy: 0.4694
454/1563 [=======>......................] - ETA: 1s - loss: 1.4891 - accuracy: 0.4684
492/1563 [========>.....................] - ETA: 1s - loss: 1.4879 - accuracy: 0.4693
530/1563 [=========>....................] - ETA: 1s - loss: 1.4891 - accuracy: 0.4700
568/1563 [=========>....................] - ETA: 1s - loss: 1.4863 - accuracy: 0.4717
606/1563 [==========>...................] - ETA: 1s - loss: 1.4886 - accuracy: 0.4708
644/1563 [===========>..................] - ETA: 1s - loss: 1.4897 - accuracy: 0.4703
682/1563 [============>.................] - ETA: 1s - loss: 1.4898 - accuracy: 0.4703
720/1563 [============>.................] - ETA: 1s - loss: 1.4859 - accuracy: 0.4717
758/1563 [=============>................] - ETA: 1s - loss: 1.4863 - accuracy: 0.4719
796/1563 [==============>...............] - ETA: 1s - loss: 1.4857 - accuracy: 0.4728
834/1563 [===============>..............] - ETA: 0s - loss: 1.4869 - accuracy: 0.4731
873/1563 [===============>..............] - ETA: 0s - loss: 1.4866 - accuracy: 0.4732
911/1563 [================>.............] - ETA: 0s - loss: 1.4879 - accuracy: 0.4730
950/1563 [=================>............] - ETA: 0s - loss: 1.4886 - accuracy: 0.4731
988/1563 [=================>............] - ETA: 0s - loss: 1.4862 - accuracy: 0.4744
1026/1563 [==================>...........] - ETA: 0s - loss: 1.4848 - accuracy: 0.4751
1064/1563 [===================>..........] - ETA: 0s - loss: 1.4857 - accuracy: 0.4749
1102/1563 [====================>.........] - ETA: 0s - loss: 1.4847 - accuracy: 0.4753
1140/1563 [====================>.........] - ETA: 0s - loss: 1.4846 - accuracy: 0.4752
1178/1563 [=====================>........] - ETA: 0s - loss: 1.4840 - accuracy: 0.4755
1216/1563 [======================>.......] - ETA: 0s - loss: 1.4829 - accuracy: 0.4757
1253/1563 [=======================>......] - ETA: 0s - loss: 1.4804 - accuracy: 0.4767
1291/1563 [=======================>......] - ETA: 0s - loss: 1.4795 - accuracy: 0.4771
1329/1563 [========================>.....] - ETA: 0s - loss: 1.4796 - accuracy: 0.4769
1367/1563 [=========================>....] - ETA: 0s - loss: 1.4803 - accuracy: 0.4764
1405/1563 [=========================>....] - ETA: 0s - loss: 1.4798 - accuracy: 0.4764
1443/1563 [==========================>...] - ETA: 0s - loss: 1.4805 - accuracy: 0.4760
1481/1563 [===========================>..] - ETA: 0s - loss: 1.4799 - accuracy: 0.4766
1519/1563 [============================>.] - ETA: 0s - loss: 1.4796 - accuracy: 0.4767
1557/1563 [============================>.] - ETA: 0s - loss: 1.4805 - accuracy: 0.4767
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4806 - accuracy: 0.4767
Epoch 5/10
1/1563 [..............................] - ETA: 2s - loss: 1.0800 - accuracy: 0.6250
38/1563 [..............................] - ETA: 2s - loss: 1.4162 - accuracy: 0.4877
76/1563 [>.............................] - ETA: 1s - loss: 1.4661 - accuracy: 0.4844
115/1563 [=>............................] - ETA: 1s - loss: 1.4407 - accuracy: 0.4929
154/1563 [=>............................] - ETA: 1s - loss: 1.4399 - accuracy: 0.4980
193/1563 [==>...........................] - ETA: 1s - loss: 1.4362 - accuracy: 0.4971
232/1563 [===>..........................] - ETA: 1s - loss: 1.4401 - accuracy: 0.4933
271/1563 [====>.........................] - ETA: 1s - loss: 1.4390 - accuracy: 0.4938
309/1563 [====>.........................] - ETA: 1s - loss: 1.4440 - accuracy: 0.4935
348/1563 [=====>........................] - ETA: 1s - loss: 1.4471 - accuracy: 0.4922
387/1563 [======>.......................] - ETA: 1s - loss: 1.4488 - accuracy: 0.4923
426/1563 [=======>......................] - ETA: 1s - loss: 1.4480 - accuracy: 0.4913
464/1563 [=======>......................] - ETA: 1s - loss: 1.4421 - accuracy: 0.4927
502/1563 [========>.....................] - ETA: 1s - loss: 1.4418 - accuracy: 0.4928
540/1563 [=========>....................] - ETA: 1s - loss: 1.4417 - accuracy: 0.4931
578/1563 [==========>...................] - ETA: 1s - loss: 1.4433 - accuracy: 0.4925
616/1563 [==========>...................] - ETA: 1s - loss: 1.4424 - accuracy: 0.4928
655/1563 [===========>..................] - ETA: 1s - loss: 1.4414 - accuracy: 0.4932
694/1563 [============>.................] - ETA: 1s - loss: 1.4406 - accuracy: 0.4933
733/1563 [=============>................] - ETA: 1s - loss: 1.4420 - accuracy: 0.4918
772/1563 [=============>................] - ETA: 1s - loss: 1.4408 - accuracy: 0.4930
811/1563 [==============>...............] - ETA: 0s - loss: 1.4415 - accuracy: 0.4933
850/1563 [===============>..............] - ETA: 0s - loss: 1.4415 - accuracy: 0.4937
888/1563 [================>.............] - ETA: 0s - loss: 1.4431 - accuracy: 0.4924
927/1563 [================>.............] - ETA: 0s - loss: 1.4435 - accuracy: 0.4932
966/1563 [=================>............] - ETA: 0s - loss: 1.4426 - accuracy: 0.4932
1005/1563 [==================>...........] - ETA: 0s - loss: 1.4437 - accuracy: 0.4929
1044/1563 [===================>..........] - ETA: 0s - loss: 1.4436 - accuracy: 0.4932
1083/1563 [===================>..........] - ETA: 0s - loss: 1.4418 - accuracy: 0.4938
1122/1563 [====================>.........] - ETA: 0s - loss: 1.4409 - accuracy: 0.4942
1161/1563 [=====================>........] - ETA: 0s - loss: 1.4391 - accuracy: 0.4950
1200/1563 [======================>.......] - ETA: 0s - loss: 1.4377 - accuracy: 0.4955
1239/1563 [======================>.......] - ETA: 0s - loss: 1.4374 - accuracy: 0.4952
1278/1563 [=======================>......] - ETA: 0s - loss: 1.4355 - accuracy: 0.4956
1316/1563 [========================>.....] - ETA: 0s - loss: 1.4347 - accuracy: 0.4959
1355/1563 [=========================>....] - ETA: 0s - loss: 1.4331 - accuracy: 0.4966
1393/1563 [=========================>....] - ETA: 0s - loss: 1.4329 - accuracy: 0.4968
1432/1563 [==========================>...] - ETA: 0s - loss: 1.4338 - accuracy: 0.4963
1470/1563 [===========================>..] - ETA: 0s - loss: 1.4329 - accuracy: 0.4962
1508/1563 [===========================>..] - ETA: 0s - loss: 1.4318 - accuracy: 0.4963
1545/1563 [============================>.] - ETA: 0s - loss: 1.4327 - accuracy: 0.4957
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4323 - accuracy: 0.4959
Epoch 6/10
1/1563 [..............................] - ETA: 2s - loss: 1.5037 - accuracy: 0.5312
40/1563 [..............................] - ETA: 1s - loss: 1.3964 - accuracy: 0.4977
79/1563 [>.............................] - ETA: 1s - loss: 1.4034 - accuracy: 0.4996
118/1563 [=>............................] - ETA: 1s - loss: 1.4059 - accuracy: 0.5011
157/1563 [==>...........................] - ETA: 1s - loss: 1.3985 - accuracy: 0.5040
196/1563 [==>...........................] - ETA: 1s - loss: 1.3991 - accuracy: 0.5037
234/1563 [===>..........................] - ETA: 1s - loss: 1.3994 - accuracy: 0.5041
272/1563 [====>.........................] - ETA: 1s - loss: 1.3991 - accuracy: 0.5043
310/1563 [====>.........................] - ETA: 1s - loss: 1.4032 - accuracy: 0.5032
348/1563 [=====>........................] - ETA: 1s - loss: 1.4016 - accuracy: 0.5052
386/1563 [======>.......................] - ETA: 1s - loss: 1.3996 - accuracy: 0.5065
424/1563 [=======>......................] - ETA: 1s - loss: 1.3990 - accuracy: 0.5060
462/1563 [=======>......................] - ETA: 1s - loss: 1.4014 - accuracy: 0.5049
500/1563 [========>.....................] - ETA: 1s - loss: 1.4015 - accuracy: 0.5060
538/1563 [=========>....................] - ETA: 1s - loss: 1.3975 - accuracy: 0.5067
576/1563 [==========>...................] - ETA: 1s - loss: 1.3954 - accuracy: 0.5079
614/1563 [==========>...................] - ETA: 1s - loss: 1.3964 - accuracy: 0.5079
652/1563 [===========>..................] - ETA: 1s - loss: 1.3974 - accuracy: 0.5085
690/1563 [============>.................] - ETA: 1s - loss: 1.3977 - accuracy: 0.5078
728/1563 [============>.................] - ETA: 1s - loss: 1.3961 - accuracy: 0.5091
766/1563 [=============>................] - ETA: 1s - loss: 1.3943 - accuracy: 0.5091
804/1563 [==============>...............] - ETA: 1s - loss: 1.3966 - accuracy: 0.5081
842/1563 [===============>..............] - ETA: 0s - loss: 1.3959 - accuracy: 0.5087
880/1563 [===============>..............] - ETA: 0s - loss: 1.3951 - accuracy: 0.5093
918/1563 [================>.............] - ETA: 0s - loss: 1.3946 - accuracy: 0.5096
956/1563 [=================>............] - ETA: 0s - loss: 1.3932 - accuracy: 0.5101
994/1563 [==================>...........] - ETA: 0s - loss: 1.3932 - accuracy: 0.5097
1032/1563 [==================>...........] - ETA: 0s - loss: 1.3937 - accuracy: 0.5100
1070/1563 [===================>..........] - ETA: 0s - loss: 1.3942 - accuracy: 0.5101
1108/1563 [====================>.........] - ETA: 0s - loss: 1.3951 - accuracy: 0.5100
1146/1563 [====================>.........] - ETA: 0s - loss: 1.3957 - accuracy: 0.5095
1184/1563 [=====================>........] - ETA: 0s - loss: 1.3949 - accuracy: 0.5100
1222/1563 [======================>.......] - ETA: 0s - loss: 1.3938 - accuracy: 0.5099
1259/1563 [=======================>......] - ETA: 0s - loss: 1.3929 - accuracy: 0.5100
1297/1563 [=======================>......] - ETA: 0s - loss: 1.3917 - accuracy: 0.5101
1335/1563 [========================>.....] - ETA: 0s - loss: 1.3922 - accuracy: 0.5102
1373/1563 [=========================>....] - ETA: 0s - loss: 1.3920 - accuracy: 0.5105
1411/1563 [==========================>...] - ETA: 0s - loss: 1.3914 - accuracy: 0.5111
1449/1563 [==========================>...] - ETA: 0s - loss: 1.3911 - accuracy: 0.5111
1487/1563 [===========================>..] - ETA: 0s - loss: 1.3913 - accuracy: 0.5108
1526/1563 [============================>.] - ETA: 0s - loss: 1.3910 - accuracy: 0.5110
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3896 - accuracy: 0.5116
Epoch 7/10
1/1563 [..............................] - ETA: 2s - loss: 1.5042 - accuracy: 0.4062
39/1563 [..............................] - ETA: 2s - loss: 1.3501 - accuracy: 0.5256
77/1563 [>.............................] - ETA: 1s - loss: 1.3670 - accuracy: 0.5170
116/1563 [=>............................] - ETA: 1s - loss: 1.3605 - accuracy: 0.5167
155/1563 [=>............................] - ETA: 1s - loss: 1.3566 - accuracy: 0.5214
193/1563 [==>...........................] - ETA: 1s - loss: 1.3551 - accuracy: 0.5202
231/1563 [===>..........................] - ETA: 1s - loss: 1.3540 - accuracy: 0.5222
269/1563 [====>.........................] - ETA: 1s - loss: 1.3633 - accuracy: 0.5188
307/1563 [====>.........................] - ETA: 1s - loss: 1.3682 - accuracy: 0.5154
346/1563 [=====>........................] - ETA: 1s - loss: 1.3679 - accuracy: 0.5154
385/1563 [======>.......................] - ETA: 1s - loss: 1.3635 - accuracy: 0.5188
424/1563 [=======>......................] - ETA: 1s - loss: 1.3635 - accuracy: 0.5188
462/1563 [=======>......................] - ETA: 1s - loss: 1.3620 - accuracy: 0.5196
499/1563 [========>.....................] - ETA: 1s - loss: 1.3649 - accuracy: 0.5192
537/1563 [=========>....................] - ETA: 1s - loss: 1.3622 - accuracy: 0.5203
575/1563 [==========>...................] - ETA: 1s - loss: 1.3623 - accuracy: 0.5200
613/1563 [==========>...................] - ETA: 1s - loss: 1.3607 - accuracy: 0.5207
650/1563 [===========>..................] - ETA: 1s - loss: 1.3610 - accuracy: 0.5210
688/1563 [============>.................] - ETA: 1s - loss: 1.3594 - accuracy: 0.5218
727/1563 [============>.................] - ETA: 1s - loss: 1.3577 - accuracy: 0.5230
766/1563 [=============>................] - ETA: 1s - loss: 1.3561 - accuracy: 0.5240
805/1563 [==============>...............] - ETA: 1s - loss: 1.3555 - accuracy: 0.5242
844/1563 [===============>..............] - ETA: 0s - loss: 1.3549 - accuracy: 0.5239
883/1563 [===============>..............] - ETA: 0s - loss: 1.3541 - accuracy: 0.5230
922/1563 [================>.............] - ETA: 0s - loss: 1.3553 - accuracy: 0.5220
959/1563 [=================>............] - ETA: 0s - loss: 1.3559 - accuracy: 0.5215
997/1563 [==================>...........] - ETA: 0s - loss: 1.3532 - accuracy: 0.5226
1035/1563 [==================>...........] - ETA: 0s - loss: 1.3534 - accuracy: 0.5231
1073/1563 [===================>..........] - ETA: 0s - loss: 1.3541 - accuracy: 0.5229
1111/1563 [====================>.........] - ETA: 0s - loss: 1.3552 - accuracy: 0.5225
1149/1563 [=====================>........] - ETA: 0s - loss: 1.3546 - accuracy: 0.5226
1187/1563 [=====================>........] - ETA: 0s - loss: 1.3542 - accuracy: 0.5226
1225/1563 [======================>.......] - ETA: 0s - loss: 1.3530 - accuracy: 0.5229
1263/1563 [=======================>......] - ETA: 0s - loss: 1.3534 - accuracy: 0.5226
1301/1563 [=======================>......] - ETA: 0s - loss: 1.3533 - accuracy: 0.5227
1339/1563 [========================>.....] - ETA: 0s - loss: 1.3532 - accuracy: 0.5230
1377/1563 [=========================>....] - ETA: 0s - loss: 1.3535 - accuracy: 0.5233
1415/1563 [==========================>...] - ETA: 0s - loss: 1.3539 - accuracy: 0.5234
1453/1563 [==========================>...] - ETA: 0s - loss: 1.3538 - accuracy: 0.5235
1491/1563 [===========================>..] - ETA: 0s - loss: 1.3537 - accuracy: 0.5236
1529/1563 [============================>.] - ETA: 0s - loss: 1.3525 - accuracy: 0.5238
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3524 - accuracy: 0.5241
Epoch 8/10
1/1563 [..............................] - ETA: 2s - loss: 1.1560 - accuracy: 0.6875
39/1563 [..............................] - ETA: 2s - loss: 1.2853 - accuracy: 0.5569
77/1563 [>.............................] - ETA: 1s - loss: 1.3022 - accuracy: 0.5483
115/1563 [=>............................] - ETA: 1s - loss: 1.2954 - accuracy: 0.5481
153/1563 [=>............................] - ETA: 1s - loss: 1.3080 - accuracy: 0.5417
191/1563 [==>...........................] - ETA: 1s - loss: 1.3152 - accuracy: 0.5398
229/1563 [===>..........................] - ETA: 1s - loss: 1.3145 - accuracy: 0.5370
267/1563 [====>.........................] - ETA: 1s - loss: 1.3144 - accuracy: 0.5406
305/1563 [====>.........................] - ETA: 1s - loss: 1.3080 - accuracy: 0.5417
343/1563 [=====>........................] - ETA: 1s - loss: 1.3119 - accuracy: 0.5409
381/1563 [======>.......................] - ETA: 1s - loss: 1.3135 - accuracy: 0.5404
419/1563 [=======>......................] - ETA: 1s - loss: 1.3105 - accuracy: 0.5392
457/1563 [=======>......................] - ETA: 1s - loss: 1.3110 - accuracy: 0.5388
495/1563 [========>.....................] - ETA: 1s - loss: 1.3164 - accuracy: 0.5374
533/1563 [=========>....................] - ETA: 1s - loss: 1.3155 - accuracy: 0.5371
571/1563 [=========>....................] - ETA: 1s - loss: 1.3191 - accuracy: 0.5361
609/1563 [==========>...................] - ETA: 1s - loss: 1.3180 - accuracy: 0.5375
647/1563 [===========>..................] - ETA: 1s - loss: 1.3168 - accuracy: 0.5389
685/1563 [============>.................] - ETA: 1s - loss: 1.3145 - accuracy: 0.5392
723/1563 [============>.................] - ETA: 1s - loss: 1.3136 - accuracy: 0.5395
761/1563 [=============>................] - ETA: 1s - loss: 1.3166 - accuracy: 0.5381
799/1563 [==============>...............] - ETA: 1s - loss: 1.3160 - accuracy: 0.5379
837/1563 [===============>..............] - ETA: 0s - loss: 1.3158 - accuracy: 0.5376
874/1563 [===============>..............] - ETA: 0s - loss: 1.3172 - accuracy: 0.5373
912/1563 [================>.............] - ETA: 0s - loss: 1.3187 - accuracy: 0.5366
950/1563 [=================>............] - ETA: 0s - loss: 1.3197 - accuracy: 0.5357
988/1563 [=================>............] - ETA: 0s - loss: 1.3194 - accuracy: 0.5357
1026/1563 [==================>...........] - ETA: 0s - loss: 1.3189 - accuracy: 0.5351
1064/1563 [===================>..........] - ETA: 0s - loss: 1.3186 - accuracy: 0.5352
1102/1563 [====================>.........] - ETA: 0s - loss: 1.3182 - accuracy: 0.5350
1139/1563 [====================>.........] - ETA: 0s - loss: 1.3175 - accuracy: 0.5350
1177/1563 [=====================>........] - ETA: 0s - loss: 1.3175 - accuracy: 0.5350
1215/1563 [======================>.......] - ETA: 0s - loss: 1.3175 - accuracy: 0.5349
1253/1563 [=======================>......] - ETA: 0s - loss: 1.3171 - accuracy: 0.5352
1290/1563 [=======================>......] - ETA: 0s - loss: 1.3178 - accuracy: 0.5350
1328/1563 [========================>.....] - ETA: 0s - loss: 1.3180 - accuracy: 0.5349
1366/1563 [=========================>....] - ETA: 0s - loss: 1.3159 - accuracy: 0.5356
1404/1563 [=========================>....] - ETA: 0s - loss: 1.3158 - accuracy: 0.5356
1442/1563 [==========================>...] - ETA: 0s - loss: 1.3155 - accuracy: 0.5353
1480/1563 [===========================>..] - ETA: 0s - loss: 1.3160 - accuracy: 0.5351
1518/1563 [============================>.] - ETA: 0s - loss: 1.3160 - accuracy: 0.5349
1556/1563 [============================>.] - ETA: 0s - loss: 1.3152 - accuracy: 0.5353
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3152 - accuracy: 0.5353
Epoch 9/10
1/1563 [..............................] - ETA: 2s - loss: 1.1881 - accuracy: 0.5938
39/1563 [..............................] - ETA: 2s - loss: 1.2589 - accuracy: 0.5609
77/1563 [>.............................] - ETA: 1s - loss: 1.2636 - accuracy: 0.5584
115/1563 [=>............................] - ETA: 1s - loss: 1.2748 - accuracy: 0.5497
153/1563 [=>............................] - ETA: 1s - loss: 1.2677 - accuracy: 0.5476
191/1563 [==>...........................] - ETA: 1s - loss: 1.2817 - accuracy: 0.5468
229/1563 [===>..........................] - ETA: 1s - loss: 1.2743 - accuracy: 0.5490
267/1563 [====>.........................] - ETA: 1s - loss: 1.2710 - accuracy: 0.5506
305/1563 [====>.........................] - ETA: 1s - loss: 1.2680 - accuracy: 0.5504
343/1563 [=====>........................] - ETA: 1s - loss: 1.2729 - accuracy: 0.5498
381/1563 [======>.......................] - ETA: 1s - loss: 1.2756 - accuracy: 0.5493
419/1563 [=======>......................] - ETA: 1s - loss: 1.2754 - accuracy: 0.5500
457/1563 [=======>......................] - ETA: 1s - loss: 1.2742 - accuracy: 0.5512
495/1563 [========>.....................] - ETA: 1s - loss: 1.2761 - accuracy: 0.5509
533/1563 [=========>....................] - ETA: 1s - loss: 1.2743 - accuracy: 0.5530
571/1563 [=========>....................] - ETA: 1s - loss: 1.2739 - accuracy: 0.5533
610/1563 [==========>...................] - ETA: 1s - loss: 1.2758 - accuracy: 0.5519
648/1563 [===========>..................] - ETA: 1s - loss: 1.2777 - accuracy: 0.5518
686/1563 [============>.................] - ETA: 1s - loss: 1.2776 - accuracy: 0.5518
724/1563 [============>.................] - ETA: 1s - loss: 1.2784 - accuracy: 0.5507
762/1563 [=============>................] - ETA: 1s - loss: 1.2801 - accuracy: 0.5493
800/1563 [==============>...............] - ETA: 1s - loss: 1.2788 - accuracy: 0.5496
838/1563 [===============>..............] - ETA: 0s - loss: 1.2771 - accuracy: 0.5508
876/1563 [===============>..............] - ETA: 0s - loss: 1.2774 - accuracy: 0.5514
914/1563 [================>.............] - ETA: 0s - loss: 1.2781 - accuracy: 0.5515
952/1563 [=================>............] - ETA: 0s - loss: 1.2781 - accuracy: 0.5514
990/1563 [==================>...........] - ETA: 0s - loss: 1.2776 - accuracy: 0.5520
1028/1563 [==================>...........] - ETA: 0s - loss: 1.2784 - accuracy: 0.5517
1066/1563 [===================>..........] - ETA: 0s - loss: 1.2798 - accuracy: 0.5505
1104/1563 [====================>.........] - ETA: 0s - loss: 1.2814 - accuracy: 0.5503
1142/1563 [====================>.........] - ETA: 0s - loss: 1.2812 - accuracy: 0.5508
1180/1563 [=====================>........] - ETA: 0s - loss: 1.2821 - accuracy: 0.5505
1218/1563 [======================>.......] - ETA: 0s - loss: 1.2804 - accuracy: 0.5512
1256/1563 [=======================>......] - ETA: 0s - loss: 1.2793 - accuracy: 0.5510
1294/1563 [=======================>......] - ETA: 0s - loss: 1.2791 - accuracy: 0.5513
1332/1563 [========================>.....] - ETA: 0s - loss: 1.2782 - accuracy: 0.5520
1370/1563 [=========================>....] - ETA: 0s - loss: 1.2810 - accuracy: 0.5512
1408/1563 [==========================>...] - ETA: 0s - loss: 1.2824 - accuracy: 0.5508
1445/1563 [==========================>...] - ETA: 0s - loss: 1.2833 - accuracy: 0.5507
1483/1563 [===========================>..] - ETA: 0s - loss: 1.2842 - accuracy: 0.5504
1521/1563 [============================>.] - ETA: 0s - loss: 1.2843 - accuracy: 0.5503
1559/1563 [============================>.] - ETA: 0s - loss: 1.2839 - accuracy: 0.5502
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2836 - accuracy: 0.5504
Epoch 10/10
1/1563 [..............................] - ETA: 2s - loss: 1.1894 - accuracy: 0.5312
39/1563 [..............................] - ETA: 2s - loss: 1.1852 - accuracy: 0.5817
77/1563 [>.............................] - ETA: 1s - loss: 1.2167 - accuracy: 0.5787
115/1563 [=>............................] - ETA: 1s - loss: 1.2289 - accuracy: 0.5717
153/1563 [=>............................] - ETA: 1s - loss: 1.2337 - accuracy: 0.5699
191/1563 [==>...........................] - ETA: 1s - loss: 1.2442 - accuracy: 0.5653
229/1563 [===>..........................] - ETA: 1s - loss: 1.2419 - accuracy: 0.5681
267/1563 [====>.........................] - ETA: 1s - loss: 1.2499 - accuracy: 0.5648
305/1563 [====>.........................] - ETA: 1s - loss: 1.2481 - accuracy: 0.5654
343/1563 [=====>........................] - ETA: 1s - loss: 1.2481 - accuracy: 0.5639
381/1563 [======>.......................] - ETA: 1s - loss: 1.2497 - accuracy: 0.5622
419/1563 [=======>......................] - ETA: 1s - loss: 1.2461 - accuracy: 0.5626
457/1563 [=======>......................] - ETA: 1s - loss: 1.2430 - accuracy: 0.5643
494/1563 [========>.....................] - ETA: 1s - loss: 1.2459 - accuracy: 0.5623
531/1563 [=========>....................] - ETA: 1s - loss: 1.2492 - accuracy: 0.5604
569/1563 [=========>....................] - ETA: 1s - loss: 1.2509 - accuracy: 0.5603
607/1563 [==========>...................] - ETA: 1s - loss: 1.2501 - accuracy: 0.5613
645/1563 [===========>..................] - ETA: 1s - loss: 1.2491 - accuracy: 0.5615
683/1563 [============>.................] - ETA: 1s - loss: 1.2475 - accuracy: 0.5625
721/1563 [============>.................] - ETA: 1s - loss: 1.2481 - accuracy: 0.5622
759/1563 [=============>................] - ETA: 1s - loss: 1.2506 - accuracy: 0.5607
797/1563 [==============>...............] - ETA: 1s - loss: 1.2497 - accuracy: 0.5610
835/1563 [===============>..............] - ETA: 0s - loss: 1.2504 - accuracy: 0.5605
873/1563 [===============>..............] - ETA: 0s - loss: 1.2514 - accuracy: 0.5597
911/1563 [================>.............] - ETA: 0s - loss: 1.2525 - accuracy: 0.5597
949/1563 [=================>............] - ETA: 0s - loss: 1.2519 - accuracy: 0.5602
987/1563 [=================>............] - ETA: 0s - loss: 1.2523 - accuracy: 0.5596
1025/1563 [==================>...........] - ETA: 0s - loss: 1.2515 - accuracy: 0.5599
1063/1563 [===================>..........] - ETA: 0s - loss: 1.2517 - accuracy: 0.5603
1101/1563 [====================>.........] - ETA: 0s - loss: 1.2529 - accuracy: 0.5601
1139/1563 [====================>.........] - ETA: 0s - loss: 1.2537 - accuracy: 0.5598
1177/1563 [=====================>........] - ETA: 0s - loss: 1.2545 - accuracy: 0.5591
1215/1563 [======================>.......] - ETA: 0s - loss: 1.2537 - accuracy: 0.5596
1253/1563 [=======================>......] - ETA: 0s - loss: 1.2548 - accuracy: 0.5594
1291/1563 [=======================>......] - ETA: 0s - loss: 1.2544 - accuracy: 0.5597
1328/1563 [========================>.....] - ETA: 0s - loss: 1.2535 - accuracy: 0.5600
1366/1563 [=========================>....] - ETA: 0s - loss: 1.2530 - accuracy: 0.5601
1404/1563 [=========================>....] - ETA: 0s - loss: 1.2543 - accuracy: 0.5602
1442/1563 [==========================>...] - ETA: 0s - loss: 1.2551 - accuracy: 0.5601
1480/1563 [===========================>..] - ETA: 0s - loss: 1.2544 - accuracy: 0.5603
1519/1563 [============================>.] - ETA: 0s - loss: 1.2554 - accuracy: 0.5598
1557/1563 [============================>.] - ETA: 0s - loss: 1.2544 - accuracy: 0.5602
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2537 - accuracy: 0.5603
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten (Flatten) (None, 3072) 0
dense (Dense) (None, 3000) 9219000
dense_1 (Dense) (None, 1000) 3001000
dense_2 (Dense) (None, 10) 10010
=================================================================
Total params: 12,230,010
Trainable params: 12,230,010
Non-trainable params: 0
_________________________________________________________________
None
wall time seconds (without loading test data set)= 35.74362897872925
hydro05
[SSH] completed
[SSH] exit-status: 0
[workspace] $ /bin/sh -xe /tmp/jenkins16550735077287292567.sh
+ scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace
Recording plot data
Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt
Finished: SUCCESS
![[Jenkins]](/static/755cd4af/images/svgs/logo.svg){kind=logo}
